diff --git a/docs/automations.md b/docs/automations.md index 9e120cfce..4adc388f0 100644 --- a/docs/automations.md +++ b/docs/automations.md @@ -25,10 +25,12 @@ merged is called built. | Calling a web address: the `http_request` action, with credentials in headers kept sealed | built | | Running Lens work: the `run_workflow`, `run_pipeline` and `run_routine` actions | built | | Changing assets and writing notes: the `set_field`, `add_to_collection` and `write_note` actions | built | -| Settings → Automations: how often followers look and how long the event log is kept | built | +| Settings → Automations: how often followers look, how long the event log is kept, and how late a timer may fire | built | | A webhook address per automation trigger, and calls to it as `webhook.received` ([Webhook trigger](#webhook-trigger)) | built | +| Timers that outlast a restart and fire once across the API and the worker (`app/domain/timers.py`, [Time](#time)) | built | +| Schedule, `at` and `relative` triggers, and actions that wait ([Time](#time)) | planned | | The Timeline page with filters, grouping, review keys and new items, and Home's Latest card ([Timeline](timeline.md)) | built | -| Storing and running automations, schedules and timers, joins, approvals, variables, the studio | planned | +| Storing and running automations, joins, approvals, variables, the studio | planned | ## Contract @@ -450,6 +452,45 @@ everyone who reads the asset sees. Calendar events aren't written: Lens signs in to Google Calendar read-only, and writing needs a new consent. Meanwhile the agent can draft an event as a note. +## Time + +Time is a trigger and a step: "every weekday at 7:00", "at 9:00 on the 20th", "30 minutes before a calendar event", +"wait 2 hours, then...", and the clock behind a join's "if no reply within 2 days". All of it rests on timers. + +### Timers + +Built (`app/domain/timers.py`). A timer is one row per key in the `timer` table: when it is due, what runs then (its +kind), what that gets (its payload), and the automation it is for. Status: `waiting`, then `fired`, or `failed`, +`cancelled` or `missed`. + +```python +timers.set(db, key, at, kind, payload, automation=None) # makes it, or replaces the one with that key +timers.cancel(db, key) # True when one was waiting +timers.cancel_for(db, automation) # every waiting one of an automation (deleted, switched off) +timers.get(db, key) + +@timers.handler("join.absent") # what runs when one of this kind fires +def _(db, cfg, payload): ... # payload + late (true or false) + timer {key, at, automation} +``` + +- **Once.** A thread started with the routines' threads (so in the API's background work and in `lens worker`) looks + every 5 seconds, and at once after `events.wake()`. Each due timer is claimed with a compare-and-set update that + only one process wins, then its handler runs. A process that stops while a handler runs leaves that firing done: + a timer fires at most once, never twice. +- **Repeating.** A handler registered with `again(payload, after)` (a schedule's next time) is re-armed in the same + update that claims it, so a repeating timer is never lost; its row keeps waiting, with `last` saying how the last + firing went. +- **Lens was off.** A timer due within `automations.catch_up` (1 hour; Settings → Automations) fires late, with + `late: true`; an older one is `missed` and counted on its automation (`missed`). A repeating one that missed + several times fires once, for its latest time if that is within the window, then waits for its next. A timer fired + within 2 minutes of its time is on time. +- **Kept small.** An automation has at most 1000 timers waiting; setting more is refused, saying so. Timers that + ended are dropped after 7 days, by a tidy `routines.run_due` calls (once an hour). A kind nothing handles fails + with a note. +- **Light.** No timer exists until an automation uses time. While none waits the thread doesn't query the table: it + checks once a minute, and at once when this process sets one. A timer set by the other process (the API, when the + worker fires them) can so fire up to a minute late. + ## Refine later - More operators: `in`, `matches` (bounded regular expressions), `between`, times against now, `changed`. @@ -486,6 +527,8 @@ the agent can draft an event as a note. - Calling a web address: secrets in the address itself (a Slack or Discord webhook URL) sealed like headers; PATCH and DELETE (DELETE would count as deleting); OAuth sign-in for an API; form-encoded bodies; retries with backoff; networks allowed for automations apart from notifications. +- Time: sub-second timers woken by `events.wake()` alone; telling the other process (the worker) at once when the + API sets a timer, rather than within a minute. - Running Lens work: a routine run's id passed on (today the scheduler starts it later); waiting for the job and passing on its outputs; roles on collections counting for `process`; running a pipeline on every asset a join collected. - Changing assets: asking first instead of failing when a move would show an asset to new people (risk that reads the diff --git a/docs/configuration.md b/docs/configuration.md index 50e1ab71f..3297c8eb4 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -463,12 +463,14 @@ Where namespaces may send notifications ([Notifications](notifications.md)). Set ## Automations -The stream of activities automations follow ([Automations](automations.md#contract)). Settings → Automations: +The stream of activities automations follow ([Automations](automations.md#contract)), and their timers. Settings → +Automations: | Setting | Default | | |---|---|---| | `events.poll_seconds` | 5 | how often followers look for new activities when nothing wakes them, 1–3600 | | `events.keep_days` | 30 | how long activities with no row of their own (the event log) are kept, 1–3650 days | +| `automations.catch_up` | `1h` | how late a timer (a schedule, an action that waits) may still fire after Lens was off, `0s` to `7d`; older ones are missed ([Time](automations.md#time)) | ## Trusted proxies diff --git a/docs/database.md b/docs/database.md index 3194f56e6..2fb49e085 100644 --- a/docs/database.md +++ b/docs/database.md @@ -75,7 +75,9 @@ of the stream keeps where its next look starts in `event_cursor:` (a p it handles in `event_claim:`; a claim is a `CREATE` only one process wins, and claims go after a day. Each person's review marks on the timeline ([Timeline](timeline.md#review-marks)) are `event_mark:`, a row only while one of its flags is set. An automation trigger's -webhook is `automation_hook:` (its automation and trigger, the hash of its token, its signing secret sealed). So that it can be read +webhook is `automation_hook:` (its automation and trigger, the hash of its token, its signing secret sealed). +Automations' timers ([Time](automations.md#time)) are `timer:` (by `status` and `at`, and by +`automation`); one that ended goes after 7 days. So that it can be read without opening files, a resource made from an email keeps the message's addressing as it was listed (`recording.mail`) and one made from a calendar event its times, place and people (`recording.event`); both are optional, and older resources simply don't have them. diff --git a/fastapi_backend/app/domain/ops_tools.py b/fastapi_backend/app/domain/ops_tools.py index 982a47e71..098bea380 100644 --- a/fastapi_backend/app/domain/ops_tools.py +++ b/fastapi_backend/app/domain/ops_tools.py @@ -33,6 +33,7 @@ "reports", "notifications", "events", + "automations", "workers", "decisions", "voice", diff --git a/fastapi_backend/app/domain/routines.py b/fastapi_backend/app/domain/routines.py index 397b5d81b..a3fb6d172 100644 --- a/fastapi_backend/app/domain/routines.py +++ b/fastapi_backend/app/domain/routines.py @@ -40,6 +40,7 @@ sources, store, telemetry, + timers, workflows, ) @@ -609,6 +610,11 @@ def run_due(db, cfg, log=print, now=None): except Exception as e: # noqa: BLE001 - the schedule carries on if log: log(f"events: {type(e).__name__}: {e}") + try: + timers.tidy(db, cfg) # timers that ended more than timers.KEEP_DAYS ago, once an hour + except Exception as e: # noqa: BLE001 - the schedule carries on + if log: + log(f"timers: {type(e).__name__}: {e}") try: analytics.tick(db, cfg, now) # once a day, while Google Analytics is on (docs/analytics.md) except Exception as e: # noqa: BLE001 - the schedule carries on @@ -651,8 +657,9 @@ def run_due(db, cfg, log=print, now=None): def start(db, cfg_fn, stop, log=None): """The scheduling threads, until `stop` is set: watched folders scanned when due (every sources.check_seconds), - routines run when due (every CHECK_SECONDS) and the scheduled backup taken when due. The API runs them with its background work; `lens worker` runs them too, - since Docker and the packages run the API without it.""" + routines run when due (every CHECK_SECONDS), the scheduled backup taken when due, and automations' timers fired + when due (timers.py). The API runs them with its background work; `lens worker` runs them too, since Docker and + the packages run the API without it.""" def every(seconds, fn, what): def loop(): @@ -673,6 +680,7 @@ def loop(): every(lambda: cfg_fn()["fedora"]["sync_seconds"], fedora.sync_due, "fedora"), every(lambda: notebook.QUIET_SECONDS / 2, notebook.organise_due, "notes"), every(lambda: CHECK_SECONDS, backups.run_due, "backups"), + timers.start(db, cfg_fn, stop, log=log), ] diff --git a/fastapi_backend/app/domain/settings.py b/fastapi_backend/app/domain/settings.py index 16270ea6f..b3dd92a66 100644 --- a/fastapi_backend/app/domain/settings.py +++ b/fastapi_backend/app/domain/settings.py @@ -56,6 +56,7 @@ "updates": ("check", "check_hours", "auto_install", "email"), "activity": None, "events": None, + "automations": None, "backups": None, "encryption": None, "fedora": None, @@ -479,6 +480,8 @@ def _check(section, key, value, default): return _activity_setting(key, value) if section == "events": return _events_setting(key, value) + if section == "automations": + return _automations_setting(key, value) if section == "backups": return _backup_setting(key, value) if section == "fedora": @@ -1028,6 +1031,25 @@ def _events_setting(key, value): return value +CATCH_UP_MAX = 7 * 86400 + + +def _automations_setting(key, value): + """Automations (docs/automations.md): catch_up is a duration like 1h, from 0s (a timer missed while Lens was off + is missed) to 7d.""" + from . import automation_spec + + if key != "catch_up": + raise ValueError(f"unknown setting automations.{key}") + try: + n = automation_spec.check_duration(value, what="automations.catch_up") + except ValueError: + n = -1 + if not 0 <= n <= CATCH_UP_MAX: + raise ValueError("automations.catch_up is a duration from 0s to 7d, like 1h") + return value.strip() + + def _backup_setting(key, value): from . import schedule diff --git a/fastapi_backend/app/domain/store.py b/fastapi_backend/app/domain/store.py index 7bde98726..b8b38c7f1 100644 --- a/fastapi_backend/app/domain/store.py +++ b/fastapi_backend/app/domain/store.py @@ -622,6 +622,9 @@ def labels(): # the stream of activities automations follow (docs/automations.md): keep_days: how long activities with no row # of their own (the event log) are kept; poll_seconds: how often followers look when nothing wakes them "events": {"keep_days": 30, "poll_seconds": 5}, + # automations (docs/automations.md): catch_up: how late a timer (a schedule, a waiting action) may still fire + # after Lens was off; older ones are missed + "automations": {"catch_up": "1h"}, # database backups (docs/database.md, Backups): schedule is a cron expression in timezone, or none (the default) # for none; the newest `keep` scheduled backups are kept. connection: a storage connection (sources.py) every new # backup is also copied to, under folder, encrypted with a key from secret.key first (none: kept here only) @@ -1268,6 +1271,11 @@ def close(self): "DEFINE TABLE IF NOT EXISTS automation_hook SCHEMALESS", "DEFINE INDEX IF NOT EXISTS automation_hook_token ON automation_hook FIELDS token_hash UNIQUE", "DEFINE INDEX IF NOT EXISTS automation_hook_automation ON automation_hook FIELDS automation", + # timers (domain/timers.py, docs/automations.md, Time): timer:, what runs when, for which automation + "DEFINE TABLE IF NOT EXISTS timer SCHEMALESS", + "DEFINE INDEX IF NOT EXISTS timer_due ON timer FIELDS status, at", + "DEFINE INDEX IF NOT EXISTS timer_automation ON timer FIELDS automation, status", + "DEFINE INDEX IF NOT EXISTS timer_done ON timer FIELDS done_at", "DEFINE INDEX IF NOT EXISTS upload_recording ON upload FIELDS recording", # review marks on the timeline (domain/event_marks.py, docs/timeline.md): each person's reviewed, important and # hidden flags on an activity, event_mark: (the id keeps the pair unique) diff --git a/fastapi_backend/app/domain/timers.py b/fastapi_backend/app/domain/timers.py new file mode 100644 index 000000000..3c8b6f75e --- /dev/null +++ b/fastapi_backend/app/domain/timers.py @@ -0,0 +1,366 @@ +"""Timers (docs/automations.md, Time): moments when something runs, kept in the database so they outlast a restart. +Automations' schedules, their `at` and `relative` triggers, actions that wait and joins that wait all use them. + +A timer is one row per key (`timer:`): when it is due (`at`, UTC), what runs then (`kind`, a function +registered with `@handler(kind)`), what that gets (`payload`), and the automation it is for. `set` makes one, or +replaces the one with that key; `cancel` and `cancel_for` call them off. A thread (`start`, started with the +routines' threads, so in the API's background work and in `lens worker`) looks every CHECK_SECONDS, and at once after +`events.wake()`, and fires what is due (`due`). Each due timer is claimed with a compare-and-set update that only one +process wins, then its handler runs, so a timer fires once even with the API and the worker both looking. A handler +that repeats (`again`: a schedule's next time) is re-armed in that same update, so a repeating timer is never lost; a +process that dies while a handler runs leaves that one firing done (at most once, never twice). + +While Lens was off: a timer due within automations.catch_up (1 hour by default) fires late, with `late: true` in its +payload; an older one is missed (counted on its automation). A repeating one fires once, for its latest time in the +window, and then waits for its next. Fired, failed, cancelled and missed rows are dropped after KEEP_DAYS (`tidy`, +which routines.run_due calls; once an hour). Nothing looks at the table until a timer is set: while none waits, the +thread doesn't query it (it checks again once a minute, or at once when this process sets one). +""" + +from __future__ import annotations + +import datetime as dt +import hashlib +import json +import logging +import os +import re +import socket +import threading +import time +import uuid +from collections.abc import Callable +from dataclasses import dataclass + +from . import automation_spec, events, store + +R = store.R +KEY_RX = re.compile(r"^[A-Za-z0-9_.:|/@+-]{1,300}$") +KIND_RX = re.compile(r"^[a-z][a-z0-9_]{0,40}(\.[a-z][a-z0-9_]{0,40}){1,2}$") +STATUSES = ("waiting", "fired", "failed", "cancelled", "missed") +ENDED = ("fired", "failed", "cancelled", "missed") +CHECK_SECONDS = 5 # how often the thread looks when nothing wakes it +IDLE_SECONDS = 60 # how long "no timer waits" is believed before the table is looked at again +LATE_SECONDS = 120 # a timer fired later than this says so (late: true); within it, it is on time +BATCH = 100 # timers fired per look at most +MAX_WAITING = 1000 # waiting timers per automation at most +MAX_PAYLOAD = 64_000 # bytes of JSON a timer's payload holds at most +MAX_REPEATS = 1000 # times of a repeating timer stepped through when looking for its latest missed one +KEEP_DAYS = 7 +SWEEP_SECONDS = 3600 +CATCH_UP = "1h" +FIELDS = "key, at, kind, payload, automation, status, fired_at, done_at, claimed_by, note, last, runs, created_at" + +log = logging.getLogger("lens.automations") +_ME = f"{socket.gethostname()}-{os.getpid()}" +_idle = {"until": 0.0} # monotonic time until which no timer is believed to wait +_swept = {"at": 0.0} +_lock = threading.Lock() + + +@dataclass(frozen=True) +class Handler: + """What runs when a timer of one kind fires: `fn(db, cfg, payload)`. `again(payload, after)` gives a repeating + timer's next time after `after` (None: it doesn't repeat, or not any more). `ended(db, payload, status, why)` is + told when one ends without firing (cancelled or missed), for what it stood for to say so.""" + + kind: str + fn: Callable[..., object] + again: Callable[[dict, dt.datetime], dt.datetime | None] | None = None + ended: Callable[..., object] | None = None + + +_HANDLERS: dict[str, Handler] = {} + + +def handler(kind, again=None, ended=None): + """Register what runs when a timer of `kind` fires (a decorator): `@timers.handler("join.absent")` over + `def _(db, cfg, payload)`. The payload is the one the timer was set with, plus `late` (fired later than + LATE_SECONDS after its time) and `timer` ({key, at, automation}). Registering a kind again replaces it.""" + if not KIND_RX.match(kind or ""): + raise ValueError(f"a timer's kind is named like schedule.next, not {kind}") + + def register(fn): + _HANDLERS[kind] = Handler(kind, fn, again, ended) + return fn + + return register + + +def handlers(): + return sorted(_HANDLERS) + + +# ---------- times ---------- +def _utc(t): + """An aware UTC datetime from a datetime (naive is UTC) or an ISO 8601 string; ValueError otherwise.""" + if isinstance(t, dt.datetime): + d = t + elif isinstance(t, str) and t.strip(): + s = t.strip() + d = dt.datetime.fromisoformat(s[:-1] + "+00:00" if s.endswith("Z") else s) + else: + raise ValueError(f"{t!r} isn't a time") + return (d if d.tzinfo else d.replace(tzinfo=dt.timezone.utc)).astimezone(dt.timezone.utc) + + +def _iso(t): + """A time as timers keep it: to the second, in UTC (2026-10-16T07:30:00+00:00), so times sort as text.""" + return _utc(t).replace(microsecond=0).isoformat(timespec="seconds") + + +def _now(now=None): + return _utc(now) if now is not None else dt.datetime.now(dt.timezone.utc) + + +def catch_up(cfg): + """Seconds a timer may be late and still fire, after Lens was off (automations.catch_up).""" + text = ((cfg or {}).get("automations") or {}).get("catch_up") or CATCH_UP + try: + return automation_spec.check_duration(text, what="automations.catch_up") + except ValueError: + return automation_spec.check_duration(CATCH_UP) + + +# ---------- setting and calling off ---------- +def _ref(key): + return R("timer", hashlib.sha1(key.encode()).hexdigest()) + + +def _check_key(key): + if not isinstance(key, str) or not KEY_RX.match(key): + raise ValueError("a timer's key is 1 to 300 letters, digits and _ . : | / @ + -, like sched-4-morning") + return key + + +def _view(r): + return {k: r.get(k) for k in ("key", "at", "kind", "payload", "automation", "status", "fired_at", "note", "last", "runs")} + + +def get(db, key): + """One timer by its key, or None.""" + r = db.one(f"SELECT {FIELDS} FROM $r", r=_ref(_check_key(key))) + return _view(r) if r else None + + +def _waiting_for(db, automation): + row = db.one("SELECT count() AS n FROM timer WHERE automation = $a AND status = 'waiting' GROUP ALL", a=automation) + return int((row or {}).get("n") or 0) + + +def set(db, key, at, kind, payload=None, automation=None): # noqa: A001 - timers.set, as the contract names it + """Make the timer `key` due at `at` (a datetime or ISO 8601 time) to run `kind`'s handler with `payload`, or + replace the one with that key, whatever it was doing (waiting, fired, ...). `automation` is the automation it is + for, which may have at most MAX_WAITING timers waiting. The timer, as `get` gives it. ValueError naming the + problem.""" + _check_key(key) + if not KIND_RX.match(kind or ""): + raise ValueError(f"a timer's kind is named like schedule.next, not {kind}") + try: + when = _iso(at) + except (TypeError, ValueError): + raise ValueError(f"timer {key}: at is a time like 2026-10-16T07:30:00Z") from None + payload = dict(payload or {}) + try: + size = len(json.dumps(payload)) + except (TypeError, ValueError): + raise ValueError(f"timer {key}: its payload is plain JSON") from None + if size > MAX_PAYLOAD: + raise ValueError(f"timer {key}: its payload is at most {MAX_PAYLOAD // 1000} KB") + if automation is not None: + if not isinstance(automation, int) or isinstance(automation, bool): + raise ValueError(f"timer {key}: automation is an automation's id") + old = db.one("SELECT status FROM $r", r=_ref(key)) + if (old or {}).get("status") != "waiting" and _waiting_for(db, automation) >= MAX_WAITING: + raise ValueError(f"automation {automation} already has {MAX_WAITING} timers waiting; let some fire or switch it off and on") + t = store.now() + row = { + "key": key, + "at": when, + "kind": kind, + "payload": payload, + "automation": automation, + "status": "waiting", + "created_at": t, + "runs": 0, + } + db.q("UPSERT $r CONTENT $d", r=_ref(key), d=store.clean(row)) + _idle["until"] = 0.0 + if _utc(when) <= _now() + dt.timedelta(seconds=CHECK_SECONDS): + events.wake() # due now: the thread in this process looks at once + return _view(row) + + +def _end(db, r, status, why, now=None): + """End a waiting timer without firing it (cancelled, missed): True when this call did. Its handler is told.""" + got = db.rows( + "UPDATE $r SET status = $s, note = $n, done_at = $t WHERE status = 'waiting' AND at = $at RETURN AFTER", + r=_ref(r["key"]), + s=status, + n=why, + t=_iso(_now(now)), + at=r["at"], + ) + if not got: + return False + h = _HANDLERS.get(r.get("kind")) + if h and h.ended: + try: + h.ended(db, dict(r.get("payload") or {}), status, why) + except Exception as e: # noqa: BLE001 - the timer is ended either way + log.warning("timer %s: %s: %s", r["key"], type(e).__name__, e) + return True + + +def cancel(db, key, why="cancelled"): + """Call off the waiting timer `key`: True when there was one.""" + r = db.one(f"SELECT {FIELDS} FROM $r WHERE status = 'waiting'", r=_ref(_check_key(key))) + return bool(r) and _end(db, r, "cancelled", why) + + +def cancel_for(db, automation, why="the automation was switched off", kinds=None, prefix=None): + """Call off every waiting timer of an automation (when it is deleted or switched off), or only those of `kinds` + or whose key starts with `prefix`: how many.""" + rows = db.rows(f"SELECT {FIELDS} FROM timer WHERE automation = $a AND status = 'waiting'", a=automation) + n = 0 + for r in rows: + if (kinds is None or r.get("kind") in kinds) and (prefix is None or str(r.get("key")).startswith(prefix)): + n += _end(db, r, "cancelled", why) + return n + + +# ---------- firing what is due ---------- +def _count_missed(db, automation): + """A missed timer counts on its automation (`missed`), when there is one.""" + if automation is None: + return + try: + db.q("UPDATE $r SET missed += 1", r=R("automation", int(automation))) + except Exception: # noqa: BLE001 - automations aren't stored yet: the timer row still says missed + pass + + +def _quiet(db): + """Whether no timer waits (believed for IDLE_SECONDS after the table last said so).""" + if time.monotonic() < _idle["until"]: + return True + if db.values("SELECT VALUE id FROM timer WHERE status = 'waiting' LIMIT 1"): + return False + _idle["until"] = time.monotonic() + IDLE_SECONDS + return True + + +def _latest(h, payload, at, now, window): + """For a repeating timer due at `at`: (its latest time at or before now that is within the window, or None; its + next time after now).""" + start = now - dt.timedelta(seconds=window) + t = at if at >= start else h.again(payload, start - dt.timedelta(microseconds=1)) + latest, n = None, 0 + while t is not None and t <= now and n < MAX_REPEATS: + latest, t, n = t, h.again(payload, t), n + 1 + nxt = t if t is not None and t > now else h.again(payload, now) + return latest, nxt + + +def _fire(db, cfg, r, now, log_fn=None): + """Fire one due timer, unless another process got it first: True when this one fired it.""" + h = _HANDLERS.get(r.get("kind")) + if h is None: + return _end(db, r, "failed", f"nothing handles timers of kind {r.get('kind')}", now) and False + payload = dict(r.get("payload") or {}) + at = _utc(r["at"]) + window = max(catch_up(cfg), LATE_SECONDS) + nxt = None + if h.again is not None: + try: + when, nxt = _latest(h, payload, at, now, window) + except Exception as e: # noqa: BLE001 - a schedule that can't be read any more ends here + return _end(db, r, "failed", f"its next time: {type(e).__name__}: {e}"[:300], now) and False + else: + when = at if now - at <= dt.timedelta(seconds=window) else None + if when is None: # missed while Lens was off + if nxt is None: + if _end(db, r, "missed", "Lens was off when it was due", now): + _count_missed(db, r.get("automation")) + return False + got = db.rows( + "UPDATE $r SET at = $nxt, note = $n WHERE status = 'waiting' AND at = $at RETURN AFTER", + r=_ref(r["key"]), + nxt=_iso(nxt), + n=f"missed {_iso(at)}: Lens was off", + at=r["at"], + ) + if got: + _count_missed(db, r.get("automation")) + return False + token = uuid.uuid4().hex + stamp = _iso(now) + got = db.rows( + "UPDATE $r SET status = $s, at = $next, fired_at = $t, done_at = $done, claim = $c, claimed_by = $me, runs += 1 " + "WHERE status = 'waiting' AND at = $at RETURN AFTER", + r=_ref(r["key"]), + s="waiting" if nxt is not None else "fired", + next=_iso(nxt) if nxt is not None else r["at"], + t=stamp, + done=None if nxt is not None else stamp, + c=token, + me=_ME, + at=r["at"], + ) + if not got: + return False + late = now - when > dt.timedelta(seconds=LATE_SECONDS) + call = {**payload, "late": late, "timer": {"key": r["key"], "at": _iso(when), "automation": r.get("automation")}} + error = None + try: + h.fn(db, cfg, call) + except Exception as e: # noqa: BLE001 - recorded on the timer; the others still fire + error = f"{type(e).__name__}: {e}"[:500] + if log_fn: + log_fn(f"timer {r['key']}: {error}") + last = store.clean({"at": _iso(when), "fired_at": stamp, "late": late, "ok": error is None, "error": error}) + db.q("UPDATE $r SET last = $l WHERE claim = $c", r=_ref(r["key"]), l=last, c=token) + if error and nxt is None: + db.q("UPDATE $r SET status = 'failed', note = $e WHERE claim = $c AND status = 'fired'", r=_ref(r["key"]), e=error, c=token) + return True + + +def due(db, cfg, now=None, log=None, limit=BATCH): + """Fire every timer that is due, in the order they were due: how many this process fired.""" + if _quiet(db): + return 0 + t = _now(now) + rows = db.rows(f"SELECT {FIELDS} FROM timer WHERE status = 'waiting' AND at <= $n ORDER BY at LIMIT $l", n=_iso(t), l=int(limit)) + return sum(1 for r in rows if _fire(db, cfg, r, t, log)) + + +def start(db, cfg_fn, stop, log=None): + """The timer thread, until `stop` is set: a look every CHECK_SECONDS, and at once after events.wake().""" + + def loop(): + while not stop.is_set(): + seen = events.generation() + try: + due(db, cfg_fn(), log=log) + except Exception as e: # noqa: BLE001 - keep going + if log: + log(f"timers: {type(e).__name__}: {e}") + events.wait(CHECK_SECONDS, stop, since=seen) + + th = threading.Thread(target=loop, daemon=True, name="timers") + th.start() + return th + + +# ---------- keeping it small ---------- +def tidy(db, cfg=None, now=None): + """Drop timers that ended (fired, failed, cancelled, missed) more than KEEP_DAYS ago, at most once an hour per + process (routines.run_due calls it every 30 seconds); the cut-off, or 0 when it didn't look.""" + t = time.monotonic() + with _lock: + if now is None and _swept["at"] and t - _swept["at"] < SWEEP_SECONDS: + return 0 + _swept["at"] = t + cut = _iso(_now(now) - dt.timedelta(days=KEEP_DAYS)) + db.q("DELETE timer WHERE status IN $s AND done_at < $c", s=list(ENDED), c=cut) + return cut diff --git a/fastapi_backend/tests/domain/test_timers.py b/fastapi_backend/tests/domain/test_timers.py new file mode 100644 index 000000000..818e9bd3a --- /dev/null +++ b/fastapi_backend/tests/domain/test_timers.py @@ -0,0 +1,218 @@ +"""Timers: set, replaced and called off; fired once when due, across two processes; repeating ones re-armed; late +and missed ones after Lens was off; the per-automation cap; the tidy; and the thread.""" + +from __future__ import annotations + +import datetime as dt +import threading + +import pytest + +from app.domain import store, timers + +UTC = dt.UTC +T0 = dt.datetime(2026, 10, 16, 7, 0, tzinfo=UTC) + + +def at(minutes=0, seconds=0, days=0): + return T0 + dt.timedelta(minutes=minutes, seconds=seconds, days=days) + + +@pytest.fixture +def fired(monkeypatch): + """A handler kind test.ping that records what it was given, and test.hourly that repeats every hour.""" + got = [] + ended = [] + monkeypatch.setattr(timers, "_HANDLERS", dict(timers._HANDLERS)) + timers._idle["until"] = 0.0 + + def hourly(payload, after): + return after.replace(minute=0, second=0, microsecond=0) + dt.timedelta(hours=1) + + timers.handler("test.ping", ended=lambda db, p, status, why: ended.append((p, status, why)))(lambda db, cfg, p: got.append(p)) + timers.handler("test.hourly", again=hourly)(lambda db, cfg, p: got.append(p)) + timers.handler("test.boom")(lambda db, cfg, p: 1 / 0) + return got, ended + + +def test_a_timer_fires_once_when_due(db, cfg, fired): + got, _ = fired + t = timers.set(db, "ping-1", at(5), "test.ping", {"n": 1}, automation=4) + assert t["at"] == "2026-10-16T07:05:00+00:00" and t["status"] == "waiting" + assert timers.due(db, cfg, now=at(4, 59)) == 0 and got == [] + assert timers.due(db, cfg, now=at(5, 3)) == 1 + assert got == [{"n": 1, "late": False, "timer": {"key": "ping-1", "at": "2026-10-16T07:05:00+00:00", "automation": 4}}] + assert timers.due(db, cfg, now=at(6)) == 0 and len(got) == 1 + row = timers.get(db, "ping-1") + assert row["status"] == "fired" and row["runs"] == 1 and row["last"]["ok"] is True + + +def test_two_processes_fire_a_timer_once(db, cfg, fired): + got, _ = fired + timers.set(db, "ping-2", at(), "test.ping", {}) + rows = db.rows(f"SELECT {timers.FIELDS} FROM timer WHERE status = 'waiting'") + # the worker and the API both read it as due; only one claim wins + assert [timers._fire(db, cfg, r, at(0, 2)) for r in rows + rows] == [True, False] + done = [] + timers.set(db, "ping-3", at(), "test.ping", {}) + rows = db.rows(f"SELECT {timers.FIELDS} FROM timer WHERE status = 'waiting'") + threads = [threading.Thread(target=lambda: done.append(timers._fire(db, cfg, rows[0], at(0, 2)))) for _ in range(4)] + for th in threads: + th.start() + for th in threads: + th.join() + assert sorted(done) == [False, False, False, True] and len(got) == 2 + + +def test_set_replaces_cancel_calls_off(db, cfg, fired): + got, ended = fired + timers.set(db, "ping-4", at(5), "test.ping", {"v": 1}, automation=4) + timers.set(db, "ping-4", at(10), "test.ping", {"v": 2}, automation=4) # moved: one timer, the new time + assert db.one("SELECT count() AS n FROM timer GROUP ALL")["n"] == 1 + assert timers.due(db, cfg, now=at(6)) == 0 + assert timers.due(db, cfg, now=at(10)) == 1 and got[0]["v"] == 2 + timers.set(db, "ping-4", at(20), "test.ping", {"v": 3}, automation=4) # set again after it fired: waits again + assert timers.get(db, "ping-4")["status"] == "waiting" + assert timers.cancel(db, "ping-4") is True and timers.cancel(db, "ping-4") is False + assert timers.get(db, "ping-4")["status"] == "cancelled" and ended[-1][1:] == ("cancelled", "cancelled") + assert timers.due(db, cfg, now=at(30)) == 0 and len(got) == 1 + for n in range(3): + timers.set(db, f"ping-5-{n}", at(5), "test.ping", {}, automation=5) + timers.set(db, "ping-6", at(5), "test.ping", {}, automation=6) + assert timers.cancel_for(db, 5, prefix="ping-5-0") == 1 + assert timers.cancel_for(db, 5) == 2 and timers.cancel_for(db, 5) == 0 + assert ended[-1][1:] == ("cancelled", "the automation was switched off") + assert timers.get(db, "ping-6")["status"] == "waiting" + + +def test_what_isnt_a_timer_is_refused(db, fired): + for args, msg in ( + (("bad key!", at(), "test.ping"), "key"), + (("k", "soon", "test.ping"), "at is a time"), + (("k", at(), "Ping"), "kind"), + (("k", at(), "test.ping", {"x": object()}), "plain JSON"), + (("k", at(), "test.ping", {"x": "y" * 70_000}), "at most 64 KB"), + ): + with pytest.raises(ValueError, match=msg): + timers.set(db, *args) + + +def test_unknown_kinds_and_failures_are_noted(db, cfg, fired): + timers.set(db, "who-1", at(), "test.nobody", {}) + timers.set(db, "boom-1", at(), "test.boom", {}) + logged = [] + assert timers.due(db, cfg, now=at(0, 1), log=logged.append) == 1 # boom fired (and failed); nobody isn't handled + assert timers.get(db, "who-1")["status"] == "failed" and "nothing handles" in timers.get(db, "who-1")["note"] + boom = timers.get(db, "boom-1") + assert boom["status"] == "failed" and "ZeroDivisionError" in boom["note"] and boom["last"]["ok"] is False + assert logged and "ZeroDivisionError" in logged[0] + + +def test_a_repeating_timer_is_re_armed_as_it_is_claimed(db, cfg, fired): + got, _ = fired + timers.set(db, "hourly", at(), "test.hourly", {"h": True}, automation=4) + assert timers.due(db, cfg, now=at(0, 4)) == 1 + row = timers.get(db, "hourly") + assert row["status"] == "waiting" and row["at"] == "2026-10-16T08:00:00+00:00" and row["last"]["at"] == "2026-10-16T07:00:00+00:00" + assert timers.due(db, cfg, now=at(59)) == 0 + assert timers.due(db, cfg, now=at(60, 1)) == 1 and [p["timer"]["at"][11:16] for p in got] == ["07:00", "08:00"] + + +def test_late_and_missed_after_lens_was_off(db, cfg, fired): + got, ended = fired + db.q("CREATE $r CONTENT {name: 'School emails'}", r=store.R("automation", 4)) + timers.set(db, "late", at(), "test.ping", {}, automation=4) + timers.set(db, "old", at(-90), "test.ping", {}, automation=4) + assert timers.due(db, cfg, now=at(45)) == 1 # 45 minutes late: within the hour of catch-up + assert got[0]["late"] is True and got[0]["timer"]["key"] == "late" + old = timers.get(db, "old") + assert old["status"] == "missed" and ended[-1][1] == "missed" + assert db.one("SELECT missed FROM $r", r=store.R("automation", 4))["missed"] == 1 + # catch-up 0s: only timers within LATE_SECONDS of their time fire + cfg = {**cfg, "automations": {"catch_up": "0s"}} + timers.set(db, "soon", at(100), "test.ping", {}) + timers.set(db, "later", at(100), "test.ping", {}) + assert timers.due(db, cfg, now=at(101)) == 2 + timers.set(db, "much-later", at(110), "test.ping", {}) + assert timers.due(db, cfg, now=at(120)) == 0 and timers.get(db, "much-later")["status"] == "missed" + + +def test_a_repeating_timer_fires_its_latest_missed_time_once(db, cfg, fired): + got, _ = fired + timers.set(db, "hourly", at(), "test.hourly", {}, automation=4) + # off from 07:00 to 10:30: 07:00, 08:00, 09:00 and 10:00 were missed; 10:00 is within the hour, so it fires once + assert timers.due(db, cfg, now=at(210)) == 1 + assert [p["timer"]["at"][11:16] for p in got] == ["10:00"] and got[0]["late"] is True + assert timers.get(db, "hourly")["at"] == "2026-10-16T11:00:00+00:00" + # with ten minutes of catch-up, 11:00 is missed at 11:30 and the timer waits for 12:00 + cfg = {**cfg, "automations": {"catch_up": "10m"}} + assert timers.due(db, cfg, now=at(270)) == 0 and len(got) == 1 + row = timers.get(db, "hourly") + assert row["status"] == "waiting" and row["at"] == "2026-10-16T12:00:00+00:00" and "missed" in row["note"] + + +def test_an_automation_has_at_most_so_many_timers_waiting(db, cfg, fired, monkeypatch): + monkeypatch.setattr(timers, "MAX_WAITING", 2) + timers.set(db, "a-1", at(5), "test.ping", {}, automation=7) + timers.set(db, "a-2", at(5), "test.ping", {}, automation=7) + timers.set(db, "a-2", at(6), "test.ping", {}, automation=7) # replacing a waiting one is fine + with pytest.raises(ValueError, match="already has 2 timers waiting"): + timers.set(db, "a-3", at(5), "test.ping", {}, automation=7) + timers.set(db, "b-1", at(5), "test.ping", {}, automation=8) # another automation's count is its own + assert timers.due(db, cfg, now=at(5)) == 2 + timers.set(db, "a-3", at(9), "test.ping", {}, automation=7) + + +def test_tidy_drops_timers_that_ended_a_week_ago(db, cfg, fired): + timers.set(db, "done", at(), "test.ping", {}) + timers.set(db, "off", at(), "test.ping", {}) + timers.set(db, "waits", at(days=30), "test.ping", {}) + timers.due(db, cfg, now=at(1)) + timers.cancel(db, "off") + assert timers.tidy(db, cfg, now=at(days=6)) and timers.get(db, "done") + timers.tidy(db, cfg, now=dt.datetime.now(UTC) + dt.timedelta(days=8)) + assert timers.get(db, "done") is None and timers.get(db, "off") is None and timers.get(db, "waits") + timers._swept["at"] = 0.0 + assert timers.tidy(db, cfg) and timers.tidy(db, cfg) == 0 # once an hour + + +def test_no_query_while_no_timer_waits(db, cfg, fired, monkeypatch): + got, _ = fired + assert timers.due(db, cfg, now=at()) == 0 and timers._idle["until"] > 0 + calls = [] + real = db.rows + monkeypatch.setattr(db, "rows", lambda *a, **k: calls.append(a) or real(*a, **k)) + row = {"key": "x", "at": "2026-10-16T07:00:00+00:00", "kind": "test.ping", "status": "waiting"} + db.q("CREATE $r CONTENT $d", r=timers._ref("x"), d=row) + assert timers.due(db, cfg, now=at(1)) == 0 and calls == [] # another process's timer: seen within a minute + timers.set(db, "mine", at(), "test.ping", {}) # one set here is seen at once + assert timers.due(db, cfg, now=at(1)) == 2 and len(got) == 2 + + +def test_the_thread_fires_what_is_due_and_stops(db, cfg, fired): + got, _ = fired + stop = threading.Event() + th = timers.start(db, lambda: cfg, stop) + try: + timers.set(db, "now", dt.datetime.now(UTC), "test.ping", {"n": 9}) # due now: wakes the thread + for _ in range(100): + if got: + break + stop.wait(0.05) + finally: + stop.set() + th.join(5) + assert got and got[0]["n"] == 9 and not th.is_alive() + + +def test_catch_up_is_a_setting(app, client, db): + from tests.helpers import login, make_user + + make_user(db, "root@x.io", "root password 1", admin=True) + h = login(client, "root@x.io", "root password 1") + assert timers.catch_up(app.state.settings.current()) == 3600 + assert client.put("/api/v1/settings/automations", json={"catch_up": "15m"}, headers=h).status_code == 200 + assert timers.catch_up(app.state.settings.current()) == 900 + assert client.get("/api/v1/settings", headers=h).json()["automations"]["values"] == {"catch_up": "15m"} + for bad in ({"catch_up": "8d"}, {"catch_up": 60}, {"catch_up": "-1h"}, {"catch_up": "soon"}, {"grace": "1h"}): + assert client.put("/api/v1/settings/automations", json=bad, headers=h).status_code == 400, bad diff --git a/nextjs-frontend/__tests__/settings-find.test.tsx b/nextjs-frontend/__tests__/settings-find.test.tsx index 5260cc263..2a6b40035 100644 --- a/nextjs-frontend/__tests__/settings-find.test.tsx +++ b/nextjs-frontend/__tests__/settings-find.test.tsx @@ -78,7 +78,12 @@ describe("finding a setting", () => { const hits = findSettings("activities", true); const auto = hits.find((h) => h.section === "automations")!; expect(auto.group).toBe("System"); - expect(fieldsOf("automations").map((f) => fieldId(f))).toEqual(["events.poll_seconds", "events.keep_days"]); + expect(fieldsOf("automations").map((f) => fieldId(f))).toEqual([ + "events.poll_seconds", + "events.keep_days", + "automations.catch_up", + ]); + expect(findSettings("missed", true).some((h) => h.section === "automations")).toBe(true); }); it("finds only People for people who aren't admins", () => { diff --git a/nextjs-frontend/components/settings/bodies.tsx b/nextjs-frontend/components/settings/bodies.tsx index 292d93185..55e572fb1 100644 --- a/nextjs-frontend/components/settings/bodies.tsx +++ b/nextjs-frontend/components/settings/bodies.tsx @@ -475,6 +475,7 @@ export function SectionBody({ ctx }: { ctx: BodyCtx }) {
+

Agents read the same stream with the timeline tool (MCP and the assistant) or GET /api/v1/activities. diff --git a/nextjs-frontend/components/settings/model.ts b/nextjs-frontend/components/settings/model.ts index 44342f563..b6aa76444 100644 --- a/nextjs-frontend/components/settings/model.ts +++ b/nextjs-frontend/components/settings/model.ts @@ -393,9 +393,9 @@ export const SECTIONS: SectionSpec[] = [ { id: "automations", label: "Automations", - backend: ["events"], + backend: ["events", "automations"], description: - "The stream of activities automations follow (new assets and what else happens in Lens): how often followers look for new ones, and how long activities with no row of their own are kept.", + "The stream of activities automations follow (new assets and what else happens in Lens): how often followers look for new ones, how long activities with no row of their own are kept, and how late a timer may still fire after Lens was off.", }, { id: "startup", @@ -2697,6 +2697,22 @@ export const FIELDS: FieldSpec[] = [ max: 3650, hint: "Activities with no row of their own, such as an automation’s; new assets and other rows stay as long as they do", }, + { + section: "automations", + key: "catch_up", + label: "After Lens was off, still run timers up to", + kind: "select", + options: [ + { value: "0s", label: "Don’t: count them as missed" }, + { value: "5m", label: "5 minutes late" }, + { value: "15m", label: "15 minutes late" }, + { value: "1h", label: "1 hour late" }, + { value: "6h", label: "6 hours late" }, + { value: "1d", label: "1 day late" }, + { value: "7d", label: "7 days late" }, + ], + hint: "Schedules and actions that wait; one that missed several times runs once, and older ones count as missed", + }, // Usage analytics (Google Analytics opt-in) { section: "analytics",