diff --git a/CHANGELOG.md b/CHANGELOG.md index cb50a2e38..627a09050 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,6 +10,11 @@ The backend (`fastapi_backend`) and the frontend (`nextjs-frontend`) are version question's words. A title someone gives it is kept. `title_by` says who titled it, `POST /api/v1/chats//title` and the MCP tool `rename_chat` (without a title) have Lens title it again, and `rename_chat` with a title renames it. Settings → AI assistant → Title conversations (on) turns it off ([Titles](docs/assistant.md#titles)). +- **More tests for automations' conditions.** An If, a trigger's where and a condition node can now test that text + starts or ends with something or matches a pattern, that a value is in a list, that a number or a time is between + two, before or after a time or within a duration, falls on certain days or times of day, and whether a value changed + since the last activity. Each value is checked when it's saved, and patterns that could tie up the worker are refused + ([Operators](docs/automations.md#operators)). - **Specs from transcripts.** Turn a voice session or a meeting into a technical spec, saved as a note linked to it: the model writes the spec, then lists every named decision, every "no" and "not that, this", every constraint and every gap the speaker left to decide later, each with its quote and a link to the moment it was said; a quote the diff --git a/README.md b/README.md index d93039736..8bc7be8cd 100644 --- a/README.md +++ b/README.md @@ -147,7 +147,8 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p calendar events, files, feed posts, what people say to Lens, sensor readings, notes, graph changes, runs, budget warnings) is kept as a stream of activities in W3C Activity Streams form, which agents read through the API, MCP and the assistant, and people review on the Timeline page (and Latest on Home), marking what they've seen. Automations act - on that stream, on a schedule, at a time or when their own webhook is called: they test conditions, then tell you in + on that stream, on a schedule, at a time or when their own webhook is called: they test conditions (text, patterns, + numbers, lists, times, days and whether a value changed), then tell you in your own chat app, ask the model or the agent, use a tool, call a web address (its credentials kept sealed), run a workflow, pipeline or routine, or change assets and notes. A Classify node in a workflow sorts an asset into labels, content types or topics, acting only on a sure answer. Anything that reaches outside Lens or changes more than what diff --git a/docs/automations.md b/docs/automations.md index 2e9fb09db..59f0446e3 100644 --- a/docs/automations.md +++ b/docs/automations.md @@ -10,6 +10,7 @@ merged is called built. | Part | Status | | --- | --- | | Match trees: one condition language for triggers, an automation's If and workflow nodes | built | +| Operators for times, lists, text patterns and changes (`in`, `matches`, `within`, `changed`...) | built | | Activities, the kinds catalog, the event log, following the stream, timeline reads (`app/domain/events.py`) | built | | New assets as `asset.added` (the `recordings` source) | built | | New assets by kind: emails received, calendar events, files from storage, uploads, pages saved from the web, imports | built | @@ -328,15 +329,57 @@ M := {"all": [M, …]} | {"any": [M, …]} | {"not": M} | {"path": "a.b.0", "op" ``` - A test (`{path, op, value}`) reads the part of the value at a dotted path, list items by number - (`object.from.0`); no path tests the value itself. The tests are the workflow engine's: `exists`, `empty`, - `equals`, `not_equals`, `contains` (text in any case, or an item of a list), `gt` and `lt` (numbers). + (`object.from.0`); no path tests the value itself. The tests are in the table below. - `all` holds when every condition in it does (an empty `all` always holds), `any` when one does (it needs at least one), `not` when its condition doesn't. - Values are literals, never templates, so matching stays cheap and safe. - A tree nests at most 6 deep and holds at most 50 tests. A problem is named with where it is: `the condition, all.1.not: compare with a number`. -- `match.describe(tree)` reads a tree in words: `object.from contains “@stmarys.org” and (object.subject contains - “trip” or object.subject contains “outing”)`. +- `match.describe(tree, kind=None)` reads a tree in words: `object.from ends with “@stmarys.org” and + object.subject matches “trip” or “outing”`. With a kind, fields are named by the titles in its schema. + +#### Operators + +The first seven are the workflow engine's tests and work exactly as a condition node's always have. The others are +built in `app/domain/match.py` (`match.OPS` lists them all). When the value tested is a list, those pass when one of +its items does (as `contains` does), and `not_in` passes when none does. A value of the wrong kind, such as text for +`between` on numbers, doesn't pass; a test's own value is checked when it is saved, with the problem in words. + +| Test | Value | Passes when the value tested | +| --- | --- | --- | +| `exists`, `empty` | none | is set; is missing, empty text, `[]` or `{}` | +| `equals`, `not_equals` | anything | is (or isn't) exactly the value | +| `contains` | anything | is text holding it in any case, or a list or object holding it | +| `gt`, `lt` | a number | is a number more (less) than it | +| `in`, `not_in` | a list of up to 500 values | is (or isn't) one of them; text in any case | +| `starts_with`, `ends_with` | text (a number is read as text) | is text, or a number, starting (ending) with it, in any case | +| `matches` | a pattern of up to 200 characters, such as `trip` or `^re:` | is text holding a match, in any case | +| `between` | two numbers, or two times: `[1, 5]`, `["-1d", "now"]` | is a number, or a time, from the first to the second (both included) | +| `before`, `after` | a time: `2026-10-16T07:30`, `now`, or from now: `2h`, `-30m` | is a time before (after) it | +| `within` | a duration: `30m`, `2h`, `1d` | is a time no further than that from now, either way | +| `weekday` | days: `["sat", "sun"]`, `["monday"]` or ISO numbers (1 is Monday) | is a time on one of those days | +| `time_between` | two times of day: `["22:00", "06:00"]` | is a time of day from the first up to the second, across midnight when the second is earlier | +| `changed` | none, or a path to keep values apart by: `actor.id` | differs from the value this test saw last time (the first one seen is no change) | + +- **Times** are ISO 8601 times or dates. A time written without a zone (in the value or in what is tested) is in the + automation's time zone, and so are `weekday` and `time_between`. Durations are a whole number and `s`, `m`, `h` or + `d`; a minus points back (`-2h` is two hours ago). +- **Patterns** are regular expressions in Python's syntax, read in any case. To keep one pattern from tying up the + worker on a crafted email, a pattern that repeats a part that repeats itself (`(a+)+`), repeats alternatives that + may start alike (`(a|aa)+`) or refers back to a group (`\1`) is refused when it is saved, a pattern reads at most + the first 20,000 characters of a value, and a match that takes over 50 ms counts as no match. Compiled patterns are + cached. +- **`env`**: `match.test(tree, value, env)` reads time and state from `env`, never from globals, so tests and dry + runs can fix them: `now` (the clock when not given), `timezone` (UTC when not given) and `state`. +- **`changed`** compares with the last value the same test saw for the same automation and trigger, kept in + `automation_state` (kind `last`) by a `match.State(db, automation, alias, dry=False)` in `env["state"]`. Make one + State per activity tested, so the If and a node testing the same thing agree. With `dry` it reads and keeps nothing. + Without a State nothing has changed. With a path in `value` (`actor.id`), each sensor or sender keeps its own last + value. Like every test, it is only reached when the tests before it in an `all` pass, so `changed` compares with + the last value seen when it was reached. +- **`automation_state`** holds what automations keep between activities. Each automation keeps at most 1000 rows; past + that the oldest go and are counted (`match.dropped(db, automation)`). A `last` value is kept 30 days after it was + last seen; `match.tidy` drops expired rows once an hour, from the routine schedule. In a workflow, a condition, filter or switch case takes either one test, as before, or a tree in `match`; with a tree, `path` (when given) is the part of the value the tree tests. Graphs saved before keep running unchanged. @@ -1074,7 +1117,8 @@ held run until you pick. ## Refine later -- More operators: `in`, `matches` (bounded regular expressions), `between`, times against now, `changed`. +- Variables inside match values (`{"var": "vars.threshold"}`), once saved variables are built. +- Statistical conditions, such as a reading above its 7-day average. - The canvas shows and edits match trees on condition, filter and switch nodes (today it edits one test; a tree saved through the API is kept). - A `?format=as2` JSON-LD view and per-actor outboxes for ActivityPub; SurrealDB `DEFINE EVENT` for Update verbs that diff --git a/docs/requirements.md b/docs/requirements.md index ace1d4423..151255a83 100644 --- a/docs/requirements.md +++ b/docs/requirements.md @@ -183,6 +183,7 @@ request, to another check that proves the same thing; loosening a requirement or | Automations act on the stream, on a schedule, at a time or when their own webhook is called | [test_automation_runs.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_automation_runs.py) `test_each_new_activity_runs_each_automation_it_matches_once`, `test_a_schedule_runs_its_automation_when_its_timer_fires`, `test_a_call_to_a_webhook_runs_its_automation`; [test_timer_triggers.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_timer_triggers.py) `test_an_at_trigger_fires_once_then_reads_done` | | They test conditions, then tell you, ask the model or the agent, use a tool, call a web address with sealed credentials, run Lens work, or change assets and notes | [test_automation_runs.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_automation_runs.py) `test_the_if_limits_and_dry_runs`; [test_actions_messages.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_actions_messages.py) `test_notify_tells_me_in_my_chat_app`; [test_actions_ai.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_actions_ai.py) `test_ask_model_answers_in_text_or_as_the_fields_a_schema_names`, `test_ask_agent_works_on_what_happened_and_saves_the_conversation`, `test_call_tool_reads_as_the_person_and_changes_only_by_asking_them`; [test_actions_web.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_actions_web.py) `test_credentials_in_headers_are_sealed_to_the_automation_and_the_host_and_never_shown`; [test_actions_run.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_actions_run.py) `test_run_workflow_queues_it_on_the_asset_as_the_automation`, `test_run_pipeline_processes_the_asset_again`; [test_actions_lens.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_actions_lens.py) `test_set_field_is_kept_in_the_assets_history_by_the_automation`, `test_write_note_writes_a_page_adds_to_a_things_page_or_notes_an_asset` | | What reaches outside Lens, or changes more than what set it off, asks first: on the Approvals page or with a yes in chat; agents decline but never approve | [test_automation_spec.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_automation_spec.py) `test_risk_follows_the_effect`; [test_approvals.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_approvals.py) `test_approve_carries_the_action_out_and_the_run_records_it`, `test_agents_over_mcp_list_and_decline_but_never_approve`; [test_approvals_chat.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_approvals_chat.py) `test_a_yes_in_my_chat_decides_an_action_inside_lens`; [approvals-page.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/approvals-page.test.tsx) `groups what waits, puts the first in focus, and Enter approves it` | +| Conditions test text, patterns, numbers, lists, times, days and whether a value changed, checked as they're saved and read in words | [test_match_ops.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_match_ops.py) `test_each_test_passes_what_it_should`, `test_checks_refuse_values_a_test_cant_use`, `test_a_pattern_reads_a_long_value_only_so_far_and_gives_up_when_slow`, `test_changed_compares_with_the_last_value_seen`, `test_the_added_tests_read_in_words` | | Each automation has limits and budgets | [test_automation_runs.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_automation_runs.py) `test_the_if_limits_and_dry_runs`, `test_a_run_over_budget_is_held_for_the_persons_pick` | | Agents make, try and run automations over MCP and in the assistant, or draft one from a sentence | [test_automation_tools.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_automation_tools.py) `test_agents_save_try_and_run_automations_over_mcp`, `test_the_assistant_saves_at_once_and_asks_to_switch_on_or_run`; [test_automations.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_automations.py) `test_an_automation_is_drafted_from_a_sentence` | diff --git a/fastapi_backend/app/api/mcp_tools.py b/fastapi_backend/app/api/mcp_tools.py index 6e500e406..ef4cd83f0 100644 --- a/fastapi_backend/app/api/mcp_tools.py +++ b/fastapi_backend/app/api/mcp_tools.py @@ -2650,7 +2650,9 @@ def unsubscribe_feed(ctx: Context, feed: int) -> dict[str, Any]: Arg( "where", "string", - 'a match tree on the activity as JSON, like {"all": [{"path": "object.name", "op": "contains", "value": "trip"}]}', + 'a match tree on the activity as JSON, like {"all": [{"path": "object.name", "op": "matches", "value": "trip|outing"}]}; ' + "op is exists, empty, equals, not_equals, contains, gt, lt, in, not_in, starts_with, ends_with, matches, between, " + "before, after, within, weekday or time_between (docs/automations.md, Operators)", max_length=20000, ), Arg("unreviewed", "boolean", "only those you haven't marked reviewed"), diff --git a/fastapi_backend/app/domain/match.py b/fastapi_backend/app/domain/match.py index 75b7008a0..49a914e1b 100644 --- a/fastapi_backend/app/domain/match.py +++ b/fastapi_backend/app/domain/match.py @@ -4,23 +4,60 @@ M := {"all": [M, ...]} | {"any": [M, ...]} | {"not": M} | {"path": "a.b.0", "op": OP, "value": V} A leaf tests the part of a value at a dotted path (list items by number, as flow.dig does; no path tests the value -itself) with one of the workflow engine's tests (flow.OPS: exists, empty, equals, not_equals, contains, gt, lt), the -same way a condition node does. `all` holds when every tree in it does (an empty one always does), `any` when one -does, `not` when its tree doesn't. Values are literals: templates belong to actions, which keeps matching cheap and -safe. A tree nests at most MAX_DEPTH deep and holds at most MAX_LEAVES tests. +itself) with one of OPS: the workflow engine's tests (flow.OPS: exists, empty, equals, not_equals, contains, gt, lt), +which work as a condition node's do, and the ones added here: `in` and `not_in` (one of a list), `starts_with`, +`ends_with` and `matches` (text), `between` (numbers or times), `before`, `after` and `within` (times, against now), +`weekday` and `time_between` (a time's day and time of day), and `changed` (against the value this test saw last +time). A list passes one of the added tests when one of its items does, as `contains` does; `not_in` passes when none +does. `all` holds when every tree in it does (an empty one always does), `any` when one does, `not` when its tree +doesn't. Values are literals: templates belong to actions, which keeps matching cheap and safe. A tree nests at most +MAX_DEPTH deep and holds at most MAX_LEAVES tests. -`env` carries what tests about time and state need (`now`, `timezone`, a `state(key)` reader). None of the tests here -reads it; the operators added later do, so that tests and dry runs can fix the clock instead of reading a global one. +`env` carries what tests about time and state read, so that tests and dry runs can fix them: `now` (ISO 8601; the +clock when it isn't given), `timezone` (the automation's, for weekday, time_between and times written without a zone; +UTC when it isn't given) and `state` (a State: what `changed` compares with; without one nothing has changed). +`changed` keeps the last value it saw in `automation_state` (kind `last`), per automation, trigger and key, and `tidy` +drops what nobody has looked at for STATE_DAYS. """ from __future__ import annotations -from . import flow +import datetime as dt +import functools +import hashlib +import json +import logging +import re +import time +from re import _parser as _sre # the parser re itself uses, to check patterns before they run + +import regex + +from . import flow, store + +R = store.R +log = logging.getLogger(__name__) MAX_DEPTH = 6 MAX_LEAVES = 50 +MAX_LIST = 500 # values in an in or not_in list +MAX_TEXT = 1000 # characters in a starts_with or ends_with value +MAX_PATTERN = 200 # characters in a matches pattern +MAX_INPUT = 20_000 # characters of a value a pattern reads +PATTERN_SECONDS = 0.05 # how long a pattern may take on one value; past that it doesn't match +STATE_DAYS = 30 # a value changed compares with is kept this long after it was last seen +MAX_OPEN = 1000 # rows an automation keeps in automation_state; past that the oldest go +SWEEP_SECONDS = 3600 GROUPS = ("all", "any", "not") LEAF = {"path", "op", "value"} +ADDED = ( + "in", "not_in", "starts_with", "ends_with", "matches", "between", "before", "after", "within", "weekday", "time_between", + "changed", +) # fmt: skip +OPS = flow.OPS + ADDED +DAYS = ("mon", "tue", "wed", "thu", "fri", "sat", "sun") +DAY_NAMES = ("Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday", "Sunday") +CLOCK_RX = re.compile(r"^([01]?\d|2[0-3]):([0-5]\d)$") WORDS = { # each test in words, and its opposite (for a test under not) "exists": ("is set", "is not set"), "empty": ("is empty", "is not empty"), @@ -29,9 +66,27 @@ "contains": ("contains", "doesn't contain"), "gt": ("is more than", "is not more than"), "lt": ("is less than", "is not less than"), + "in": ("is", "is not"), + "not_in": ("is not", "is"), + "starts_with": ("starts with", "doesn't start with"), + "ends_with": ("ends with", "doesn't end with"), + "matches": ("matches", "doesn't match"), + "between": ("is between", "is not between"), + "before": ("is before", "is not before"), + "after": ("is after", "is not after"), + "within": ("is within", "is not within"), + "weekday": ("falls on a", "doesn't fall on a"), + "time_between": ("is in the day between", "is not in the day between"), + "changed": ("changed", "didn't change"), } +_swept = {"at": 0.0} + +def _num(v): + return isinstance(v, (int, float)) and not isinstance(v, bool) + +# ---------- checking ---------- def check(tree, what="the condition"): """The tree, cleaned: ValueError naming the first problem, and where it is (`what`, then the path into the tree, such as "the condition, all.1: ...").""" @@ -63,17 +118,230 @@ def _check(m, what, depth, count, at): count[0] += 1 if count[0] > MAX_LEAVES: raise ValueError(f"{what} has at most {MAX_LEAVES} tests") - problem = flow.test_problem(m) - if problem: - raise ValueError(f"{where}: {problem}") - out = {"op": m["op"]} + try: + return check_leaf(m) + except ValueError as e: + raise ValueError(f"{where}: {e}") from None + + +def check_leaf(m): + """One test ({path, op, value}), cleaned: ValueError saying what is wrong with it.""" + op = m.get("op") + if op not in OPS: + raise ValueError(f"the test is one of {', '.join(OPS)}") + if m.get("path") and not flow.PATH_RX.match(str(m["path"])): + raise ValueError("give a path like summary.importance, or none") + out = {"op": op} if m.get("path"): out["path"] = str(m["path"]) - if "value" in m: - out["value"] = m["value"] + if op in flow.OPS: + problem = flow.test_problem(m) + if problem: + raise ValueError(problem) + if "value" in m: + out["value"] = m["value"] + return out + value = _VALUES[op](m.get("value")) + if value is not None: + out["value"] = value return out +def _literal(x): + return x is None or isinstance(x, (str, int, float, bool)) + + +def _list_value(v): + if not isinstance(v, list) or not 1 <= len(v) <= MAX_LIST or not all(_literal(x) for x in v): + raise ValueError(f"give a list of 1 to {MAX_LIST} values (text, numbers, true or false)") + return list(v) + + +def _text_value(v): + if _num(v): + v = str(v) + if not isinstance(v, str) or not v or len(v) > MAX_TEXT: + raise ValueError(f"give the text to look for, up to {MAX_TEXT} characters") + return v + + +def _pattern_value(v): + if not isinstance(v, str) or not v or len(v) > MAX_PATTERN: + raise ValueError(f"give a pattern of up to {MAX_PATTERN} characters, like trip|outing") + try: + parsed = _sre.parse(v, re.IGNORECASE) + except re.error as e: + raise ValueError(f"the pattern doesn't read: {e.msg}") from None + if _refers(parsed): + raise ValueError("the pattern refers back to a group (like \\1); write it without") + if _risky(parsed): + raise ValueError( + "the pattern repeats a part that repeats itself, like (a+)+, or repeats alternatives that start alike, " + "like (a|aa)+; that can take very long, so write it more simply" + ) + try: + _rx(v) + except regex.error as e: + raise ValueError(f"the pattern doesn't read: {e}") from None + return v + + +_REPEATS = {_sre.MAX_REPEAT, _sre.MIN_REPEAT, _sre.POSSESSIVE_REPEAT} + + +def _subs(op, av): + """The parsed pieces inside one parsed piece.""" + if op in _REPEATS: + return [av[2]] + if op is _sre.SUBPATTERN: + return [av[-1]] + if op is _sre.BRANCH: + return list(av[1]) + if op in (_sre.ASSERT, _sre.ASSERT_NOT): + return [av[1]] + if op is _sre.ATOMIC_GROUP: + return [av] + if op is _sre.GROUPREF_EXISTS: + return [x for x in av[1:] if x is not None] + return [] + + +def _refers(items): + return any(op is _sre.GROUPREF or any(_refers(s) for s in _subs(op, av)) for op, av in items) + + +def _first(alt): + """The character an alternative starts with, when it is one plain character; None otherwise.""" + if not alt: + return None + op, av = alt[0] + if op is _sre.SUBPATTERN: + return _first(list(av[-1])) + return chr(av).casefold() if op is _sre.LITERAL else None + + +def _risky(items, inside=False): + """Whether a parsed pattern can backtrack without end: a part repeated more than once inside a part that is, or + alternatives inside such a part that may start alike (`(a|aa)+`).""" + for op, av in items: + if op in _REPEATS: + many = av[1] > 1 + if many and inside: + return True + if _risky(av[2], inside or many): + return True + continue + if op is _sre.BRANCH and inside: + starts = [_first(list(alt)) for alt in av[1]] + if None in starts or len(set(starts)) != len(starts): + return True + if any(_risky(s, inside) for s in _subs(op, av)): + return True + return False + + +@functools.lru_cache(maxsize=512) +def _rx(pattern): + return regex.compile(pattern, regex.IGNORECASE | regex.V0) + + +def _when_value(v): + from . import automation_spec + + if v == "now": + return v + if isinstance(v, str) and v.strip(): + try: + automation_spec.check_duration(v, back=True) + return v.strip() + except ValueError: + pass + if _time(v, dt.timezone.utc) is not None: + return v.strip() + raise ValueError("compare with a time like 2026-10-16T07:30, with now, or with a time from now like 2h (-2h: 2 hours ago)") + + +def _between_value(v): + if not isinstance(v, list) or len(v) != 2: + raise ValueError("give the two ends, like [1, 5] or [2026-10-01, 2026-10-31]") + lo, hi = v + if _num(lo) and _num(hi): + if lo > hi: + raise ValueError("the first end is the lower one") + return [lo, hi] + if _num(lo) or _num(hi): + raise ValueError("give two numbers or two times") + lo, hi = _when_value(lo), _when_value(hi) + a, b = _time(lo, dt.timezone.utc), _time(hi, dt.timezone.utc) + if a is not None and b is not None and a > b: + raise ValueError("the first end is the earlier one") + return [lo, hi] + + +def _within_value(v): + from . import automation_spec + + if not isinstance(v, str) or automation_spec.check_duration(v, what="within") <= 0: + raise ValueError("within is a time from now like 2h or 7d") + return v.strip() + + +def _day(d): + if _num(d) and int(d) == d and 1 <= d <= 7: + return DAYS[int(d) - 1] + s = str(d).strip().casefold()[:3] if isinstance(d, str) else "" + if s in DAYS: + return s + raise ValueError("give the days, like [sat, sun] or [monday, friday]") + + +def _weekday_value(v): + if not isinstance(v, list) or not 1 <= len(v) <= 7: + raise ValueError("give the days, like [sat, sun] or [monday, friday]") + return sorted({_day(d) for d in v}, key=DAYS.index) + + +def _clock(s): + m = CLOCK_RX.match(str(s).strip()) if isinstance(s, str) else None + if not m: + raise ValueError("give two times of day, like [22:00, 06:00]") + return f"{int(m.group(1)):02d}:{m.group(2)}" + + +def _time_between_value(v): + if not isinstance(v, list) or len(v) != 2: + raise ValueError("give two times of day, like [22:00, 06:00]") + a, b = _clock(v[0]), _clock(v[1]) + if a == b: + raise ValueError("give two different times of day") + return [a, b] + + +def _changed_value(v): + if v is None or v == "": + return None + if not isinstance(v, str) or not flow.PATH_RX.match(v): + raise ValueError("changed compares with the last value seen; its value, when given, is a path to keep them apart by, like actor.id") + return v + + +_VALUES = { + "in": _list_value, + "not_in": _list_value, + "starts_with": _text_value, + "ends_with": _text_value, + "matches": _pattern_value, + "between": _between_value, + "before": _when_value, + "after": _when_value, + "within": _within_value, + "weekday": _weekday_value, + "time_between": _time_between_value, + "changed": _changed_value, +} + + +# ---------- testing ---------- def test(tree, value, env=None): """Whether `value` passes the tree (checked with `check` first).""" if "all" in tree: @@ -82,9 +350,251 @@ def test(tree, value, env=None): return any(test(m, value, env) for m in tree["any"]) if "not" in tree: return not test(tree["not"], value, env) - return flow.test(tree, value) + return test_leaf(tree, value, env) + + +def test_leaf(m, value, env=None): + """Whether `value` passes one (checked) test.""" + op = m["op"] + if op in flow.OPS: + return flow.test(m, value) + v, want = flow.dig(value, m.get("path")), m.get("value") + if op == "changed": + st = state_of(env) + if st is None: + return False + key = m.get("path") or "" + if want: + key += "|" + _canon(flow.dig(value, want)) + return st.changed(key, v) + if op in ("in", "not_in"): + return _some(v, lambda x: _one_of(x, want)) == (op == "in") + if v is None: + return False + if op in ("starts_with", "ends_with"): + w = want.casefold() + if op == "starts_with": + return _some(v, lambda x: (t := _text(x)) is not None and t.casefold().startswith(w)) + return _some(v, lambda x: (t := _text(x)) is not None and t.casefold().endswith(w)) + if op == "matches": + return _some(v, lambda x: (t := _text(x)) is not None and _search(want, t)) + if op == "between" and _num(want[0]): + return _some(v, lambda x: _num(x) and want[0] <= x <= want[1]) + tz = _zone(env) + + def at(x): + return _time(x, tz) + + if op == "between": + lo, hi = when(want[0], env), when(want[1], env) + return _some(v, lambda x: (t := at(x)) is not None and lo <= t <= hi) + if op == "before": + b = when(want, env) + return _some(v, lambda x: (t := at(x)) is not None and t < b) + if op == "after": + b = when(want, env) + return _some(v, lambda x: (t := at(x)) is not None and t > b) + if op == "within": + from . import automation_spec + + n, now = automation_spec.check_duration(want), now_of(env) + return _some(v, lambda x: (t := at(x)) is not None and abs((t - now).total_seconds()) <= n) + if op == "weekday": + days = {DAYS.index(d) for d in want} + return _some(v, lambda x: (t := at(x)) is not None and t.astimezone(tz).weekday() in days) + if op == "time_between": + a, b = (int(c[:2]) * 60 + int(c[3:]) for c in want) + def inside(t): + mins = t.astimezone(tz).hour * 60 + t.astimezone(tz).minute + return a <= mins < b if a < b else mins >= a or mins < b + return _some(v, lambda x: (t := at(x)) is not None and inside(t)) + raise ValueError(f"no test {op}") + + +def _some(v, one): + """Whether a value passes: one of its items, for a list.""" + return any(one(x) for x in v) if isinstance(v, list) else one(v) + + +def _one_of(x, want): + for w in want: + if isinstance(x, str) and isinstance(w, str): + if x.casefold() == w.casefold(): + return True + elif isinstance(x, bool) == isinstance(w, bool) and x == w: + return True + return False + + +def _text(x): + if isinstance(x, str): + return x + return str(x) if _num(x) else None + + +def _search(pattern, text): + try: + return _rx(pattern).search(text[:MAX_INPUT], timeout=PATTERN_SECONDS) is not None + except TimeoutError: + log.warning("the pattern %r took over %ss on a value and counts as not matching", pattern, PATTERN_SECONDS) + return False + + +def _zone(env): + from . import schedule + + try: + return schedule.zone((env or {}).get("timezone") or "UTC") + except ValueError: + return dt.timezone.utc + + +def _time(x, tz): + """An aware time from an ISO 8601 time or date (one without a zone is in `tz`), or a datetime; None for anything + else.""" + if isinstance(x, dt.datetime): + d = x + elif isinstance(x, str) and 8 <= len(x.strip()) <= 40 and x.strip()[:4].isdigit(): + s = x.strip() + try: + d = dt.datetime.fromisoformat(s[:-1] + "+00:00" if s[-1] in "zZ" else s) + except ValueError: + return None + else: + return None + return d if d.tzinfo else d.replace(tzinfo=tz) + + +def now_of(env): + """The time a test calls now: env's `now`, else the clock.""" + n = _time((env or {}).get("now"), dt.timezone.utc) + return n or dt.datetime.now(dt.timezone.utc) + + +def when(v, env=None): + """The moment a before, after or between value names: now, a time from now (2h, -30m) or a time (in env's time + zone when it gives none).""" + from . import automation_spec + + if v == "now": + return now_of(env) + try: + return now_of(env) + dt.timedelta(seconds=automation_spec.check_duration(v, back=True)) + except ValueError: + return _time(v, _zone(env)) + + +def _canon(v): + return json.dumps(v, sort_keys=True, ensure_ascii=False, default=str) + + +# ---------- what changed compares with ---------- +class State: + """What `changed` compares with: the last value each test saw, kept per automation and trigger (alias) in + `automation_state` (kind `last`). Make one State per activity tested: within it each key is compared once, so an + automation's If and a node that test the same thing agree. With `dry` it reads and keeps nothing new; with no + database it keeps the values in `values` (a dict), for tests.""" + + def __init__(self, db=None, automation=None, alias=None, dry=False, values=None): + self.db, self.automation, self.alias, self.dry = db, automation, alias, dry + self.values = {} if values is None else values + self._said: dict[str, bool] = {} + + def changed(self, key, value): + """Whether `value` differs from the last one seen under `key` (the first one seen is no change), keeping it.""" + if key in self._said: + return self._said[key] + text = _canon(value) + if self.db is None: + before = self.values.get(key) + if not self.dry: + self.values[key] = text + else: + before = self._swap(key, text) + out = before is not None and before != text + self._said[key] = out + return out + + def _swap(self, key, text): + ref = state_ref(self.automation, "last", self.alias, key) + row = self.db.one("SELECT last FROM $r", r=ref) + if not self.dry: + if row is None: + make_room(self.db, self.automation) + t = _stamp() + self.db.q( + "UPSERT $r SET automation = $a, kind = 'last', alias = $al, key = $k, last = $v, at = $t, expires_at = $x", + r=ref, + a=self.automation, + al=self.alias, + k=str(key)[:300], + v=text, + t=t, + x=_later(STATE_DAYS * 86400), + ) + return (row or {}).get("last") + + +def state_of(env): + """The State in env (a dict is used as a State's values), or None.""" + st = (env or {}).get("state") + if isinstance(st, dict): + return State(values=st) + return st if isinstance(st, State) else None + + +def state_ref(automation, kind, alias, key): + """The automation_state row of one automation's kind of state (last, join, decide) for a trigger and key.""" + return R("automation_state", hashlib.sha1(f"{automation}|{kind}|{alias}|{key}".encode()).hexdigest()) + + +def _stamp(): + """When a state row was last written, to the microsecond: the oldest go first when an automation keeps too many.""" + return dt.datetime.now(dt.timezone.utc).isoformat(timespec="microseconds") + + +def _later(seconds): + return (dt.datetime.now(dt.timezone.utc) + dt.timedelta(seconds=seconds)).isoformat(timespec="seconds") + + +def make_room(db, automation): + """Before an automation opens one more row: past MAX_OPEN rows its oldest go, and are counted (`dropped`).""" + n = (db.one("SELECT count() AS n FROM automation_state WHERE automation = $a GROUP ALL", a=automation) or {}).get("n") or 0 + if n < MAX_OPEN: + return 0 + over = n - MAX_OPEN + 1 + old = db.rows("SELECT id, at FROM automation_state WHERE automation = $a ORDER BY at LIMIT $n", a=automation, n=over) + db.q("DELETE automation_state WHERE id IN $ids", ids=[R("automation_state", r["id"]) for r in old]) + db.q( + "UPSERT $r SET automation = $a, n += $n, at = $t", + r=R("automation_state_dropped", str(automation)), + a=automation, + n=over, + t=store.now(), + ) + return over + + +def dropped(db, automation): + """How many of an automation's rows went to make room.""" + return int((db.one("SELECT n FROM $r", r=R("automation_state_dropped", str(automation))) or {}).get("n") or 0) + + +def tidy(db, cfg=None, now=None): + """Drop automation_state rows past their expires_at, at most once an hour per process (routines.run_due calls it + every 30 seconds); the cut-off, or 0 when it didn't look.""" + t = time.monotonic() + if now is None and _swept["at"] and t - _swept["at"] < SWEEP_SECONDS: + return 0 + _swept["at"] = t + cut = now or store.now() + db.q("DELETE automation_state WHERE expires_at < $c", c=cut) + return cut + + +# ---------- reading trees ---------- def leaves(tree): """The tests in a tree, in order.""" if "all" in tree or "any" in tree: @@ -120,9 +630,60 @@ def _value(v): return str(v) +def _either(items, word="or"): + items = list(items) + return items[0] if len(items) == 1 else f"{', '.join(items[:-1])} {word} {items[-1]}" + + +UNIT_WORDS = {"s": "second", "m": "minute", "h": "hour", "d": "day"} + + +def duration_words(text): + """A duration in words: 2h is "2 hours", -30m "30 minutes" (the sign is the caller's).""" + s = str(text).strip().lstrip("-") + n, unit = int(s[:-1]), UNIT_WORDS[s[-1]] + return f"{n} {unit}{'' if n == 1 else 's'}" + + +def _when_words(v): + if v == "now": + return "now" + if isinstance(v, str) and re.match(r"^-?\d+[smhd]$", v): + return f"{duration_words(v)} ago" if v.startswith("-") else f"{duration_words(v)} from now" + return str(v) + + +def _pattern_words(p): + words = p.split("|") + if all(re.match(r"^[\w @.'-]+$", w) for w in words): + return _either([f"“{w}”" for w in words]) + return f"the pattern “{p}”" + + +def _value_words(op, v, kind): + if op in ("in", "not_in"): + return _either([_value(x) for x in v]) + if op == "matches": + return _pattern_words(v) + if op == "between": + return f"{_value(v[0]) if _num(v[0]) else _when_words(v[0])} and {_value(v[1]) if _num(v[1]) else _when_words(v[1])}" + if op in ("before", "after"): + return _when_words(v) + if op == "within": + return f"{duration_words(v)} of now" + if op == "weekday": + return _either([DAY_NAMES[DAYS.index(d)] for d in v]) + if op == "time_between": + return f"{v[0]} and {v[1]}" + if op == "changed": + return f"for each {_label(v, kind)}" if v else "" + return _value(v) + + def describe(tree, kind=None): """The tree in plain words, as the builder and the agent show it: "the sender contains “@stmarys.org” and the - subject is “Trip”". `kind` is the kind of activity the tree tests (events.py), for naming its fields.""" + subject matches “trip” or “outing”". `kind` is the kind of activity the tree tests (events.py), for naming its + fields.""" return _describe(tree, kind, True) @@ -143,4 +704,6 @@ def _describe(m, kind, top): def _leaf(m, kind, negated): said = WORDS.get(m["op"], (m["op"], f"not {m['op']}"))[1 if negated else 0] words = f"{_label(m.get('path'), kind)} {said}" - return words if m["op"] in ("exists", "empty") else f"{words} {_value(m.get('value'))}" + if m["op"] in ("exists", "empty") or (m["op"] == "changed" and not m.get("value")): + return words + return f"{words} {_value_words(m['op'], m.get('value'), kind)}" diff --git a/fastapi_backend/app/domain/routines.py b/fastapi_backend/app/domain/routines.py index 7f1291dea..45ca65d94 100644 --- a/fastapi_backend/app/domain/routines.py +++ b/fastapi_backend/app/domain/routines.py @@ -30,6 +30,7 @@ events, fedora, jobs, + match, notebook, organize, schedule, @@ -609,6 +610,7 @@ def run_due(db, cfg, log=print, now=None): activity.tidy(db, cfg) # the activity ledger past activity.keep_days, once an hour try: events.tidy(db, cfg) # the event log past events.keep_days and claims past a day, once an hour + match.tidy(db, cfg) # what automations kept between activities, past its expires_at, once an hour except Exception as e: # noqa: BLE001 - the schedule carries on if log: log(f"events: {type(e).__name__}: {e}") diff --git a/fastapi_backend/app/domain/store.py b/fastapi_backend/app/domain/store.py index 3a1333921..b074643bf 100644 --- a/fastapi_backend/app/domain/store.py +++ b/fastapi_backend/app/domain/store.py @@ -1335,6 +1335,12 @@ def close(self): "DEFINE TABLE IF NOT EXISTS event_cursor SCHEMALESS", "DEFINE TABLE IF NOT EXISTS event_claim SCHEMALESS", "DEFINE INDEX IF NOT EXISTS event_claim_at ON event_claim FIELDS at", + # what automations keep between activities (domain/match.py): the last value a `changed` test saw, joins waiting + # for their other triggers and decide answers (automation_state:), and how many went to make room + "DEFINE TABLE IF NOT EXISTS automation_state SCHEMALESS", + "DEFINE INDEX IF NOT EXISTS automation_state_automation ON automation_state FIELDS automation, at", + "DEFINE INDEX IF NOT EXISTS automation_state_expires ON automation_state FIELDS expires_at", + "DEFINE TABLE IF NOT EXISTS automation_state_dropped SCHEMALESS", # an automation trigger's webhook (domain/automation_hooks.py): automation_hook:, the token kept only as a hash "DEFINE TABLE IF NOT EXISTS automation_hook SCHEMALESS", "DEFINE INDEX IF NOT EXISTS automation_hook_token ON automation_hook FIELDS token_hash UNIQUE", diff --git a/fastapi_backend/pyproject.toml b/fastapi_backend/pyproject.toml index 556c53ca5..a7675df8f 100644 --- a/fastapi_backend/pyproject.toml +++ b/fastapi_backend/pyproject.toml @@ -26,6 +26,7 @@ dependencies = [ "opentelemetry-exporter-otlp-proto-http>=1.45.1,<2", "webauthn>=3.0.1,<4", "feedparser>=6.0.11,<7", + "regex>=2024.11", # patterns in automations' conditions, with a time limit (Apache-2.0; fastapi-mail needs it too) ] [project.optional-dependencies] diff --git a/fastapi_backend/tests/domain/test_match_ops.py b/fastapi_backend/tests/domain/test_match_ops.py new file mode 100644 index 000000000..bc07e442e --- /dev/null +++ b/fastapi_backend/tests/domain/test_match_ops.py @@ -0,0 +1,232 @@ +"""The tests match trees add to the workflow engine's (docs/automations.md, Match trees: Operators): one table per +test of what passes and what doesn't, the values each check refuses, patterns that could run without end, times read +in the automation's time zone against a fixed now, `changed` against the last value seen, and each in words.""" + +from __future__ import annotations + +import re + +import pytest + +from app.domain import match + +NOW = "2026-10-15T16:00:00Z" # a Thursday +ENV = {"now": NOW, "timezone": "Europe/London"} # 17:00 in London (summer time) + + +def leaf(path, op, value=None): + return {"path": path, "op": op, **({"value": value} if value is not None else {})} + + +def passes(op, value, got, env=ENV): + """Whether {"x": got} passes the test `op` with `value`.""" + return match.test(match.check(leaf("x", op, value)), {"x": got}, env) + + +CASES = [ # op, value, passes, doesn't + ("in", ["Draft", "sent", 3], ["draft", "SENT", 3, ["x", "sent"]], ["drafts", "3", 4, None, [], ["x"], True]), + ("in", [True, None], [True, None, [False, True]], [1, False, "true"]), + ("not_in", ["spam", "junk"], ["inbox", None, [], ["inbox", "work"], 4], ["SPAM", ["inbox", "junk"]]), + ("starts_with", "Re:", ["re: trip", "RE: x", ["Fwd: a", "Re: b"]], ["Fwd: re: x", None, 12, {"a": "Re:"}]), + ("starts_with", "+44", ["+44 7700", ["+1", "+447"]], ["0044", 44]), + ("starts_with", 44, [4412, "44 x"], [12]), + ("ends_with", "@stmarys.org", ["office@STMARYS.org", ["a@b.c", "x@stmarys.org"]], ["office@stmarys.org.uk", None]), + ("matches", "trip|outing", ["School TRIP on Friday", ["no", "an outing"]], ["Parents' evening", None, 7, []]), + ("matches", r"^\d{3}-\d{4}$", ["555-0100", ["x", "555-0100"]], ["5550100", 555]), + ("between", [1, 5], [1, 5, 2.5, [9, 3]], [0, 6, "3", None, True]), + ( + "between", + ["2026-10-01", "2026-10-31"], + ["2026-10-15T09:00:00Z", "2026-10-31", ["2026-11-02", "2026-10-02"]], + ["2026-11-01T00:00:00Z", "soon", 5], + ), + ("between", ["-1d", "now"], ["2026-10-15T15:59:59Z", "2026-10-14T16:00:00Z"], ["2026-10-15T16:00:01Z", "2026-10-14T15:59:00Z"]), + ("before", "now", ["2026-10-15T15:59:59Z", "2026-10-15T16:59:59+01:00"], ["2026-10-15T16:00:00Z", "2026-10-15T17:00:01+01:00"]), + ("before", "-2h", ["2026-10-15T13:59:00Z"], ["2026-10-15T14:00:00Z", "nonsense", None]), + ("after", "2h", ["2026-10-15T18:00:01Z", ["2026-01-01", "2027-01-01"]], ["2026-10-15T18:00:00Z"]), + ("after", "2026-10-15T17:00", ["2026-10-15T16:00:01Z"], ["2026-10-15T16:00:00Z"]), # a time with no zone: London's + ("within", "1d", ["2026-10-16T15:00:00Z", "2026-10-14T17:00:00Z", NOW], ["2026-10-16T16:00:01Z", "2026-10-14T15:59:59Z", 3]), + ("weekday", ["sat", "Sunday", 1], ["2026-10-17T12:00:00Z", "2026-10-18", "2026-10-19T08:00:00Z"], ["2026-10-15T12:00:00Z", "Saturday"]), + ("weekday", ["fri"], ["2026-10-16T22:30:00Z"], ["2026-10-16T23:30:00Z"]), # Friday 23:30 UTC is Saturday in London + ( + "time_between", + ["22:00", "6:00"], + ["2026-10-15T21:00:00Z", "2026-10-16T04:59:00Z", "2026-10-15T23:00:00+01:00"], + ["2026-10-15T20:59:00Z", "2026-10-16T05:00:00Z"], + ), + ( + "time_between", + ["09:00", "17:00"], + ["2026-10-15T15:59:00Z", "2026-10-15T08:00:00Z"], + [NOW, "2026-10-15T07:59:00Z"], + ), # from 9 to before 17 +] + + +@pytest.mark.parametrize(("op", "value", "good", "bad"), CASES, ids=[f"{c[0]}-{n}" for n, c in enumerate(CASES)]) +def test_each_test_passes_what_it_should(op, value, good, bad): + for got in good: + assert passes(op, value, got), f"{got!r} should pass {op} {value!r}" + for got in bad: + assert not passes(op, value, got), f"{got!r} shouldn't pass {op} {value!r}" + + +def test_the_added_tests_read_the_clock_and_zone_from_env(): + assert passes("time_between", ["09:00", "17:00"], NOW, {"now": NOW}) # 16:00 in UTC, the zone when none is given + assert not passes("time_between", ["09:00", "17:00"], NOW, {"timezone": "Asia/Tokyo"}) # 01:00 the next day there + assert passes("weekday", ["fri"], NOW, {"timezone": "Asia/Tokyo"}) + assert passes("within", "1h", "2026-10-15T16:30:00Z", {"now": "2026-10-15T16:00:00Z"}) + assert not passes("within", "1h", "2026-10-15T16:30:00Z", {"now": "2026-10-15T18:00:00Z"}) + assert passes("before", "now", "2000-01-01", None) # no env: the clock + assert passes("after", "2026-10-15", "2026-10-15T00:30:00Z", {"timezone": "Europe/London"}) # midnight there is 23:00 UTC + assert passes("weekday", ["thu"], NOW, {"timezone": "Nowhere/Else"}) # a zone that isn't one: UTC + + +def test_checks_refuse_values_a_test_cant_use(): + bad = [ + (leaf("x", "in", "draft"), "give a list of 1 to 500 values"), + (leaf("x", "in", []), "give a list"), + (leaf("x", "not_in", [{"a": 1}]), "give a list"), + (leaf("x", "in", list(range(501))), "give a list of 1 to 500"), + (leaf("x", "starts_with", ""), "give the text to look for"), + (leaf("x", "ends_with", ["a"]), "give the text to look for"), + (leaf("x", "matches", "x" * 201), "up to 200 characters"), + (leaf("x", "matches", "(trip"), "the pattern doesn't read: missing )"), + (leaf("x", "matches", "(a+)+$"), "repeats a part that repeats itself"), + (leaf("x", "matches", r"(\w+\s?)*$"), "repeats a part that repeats itself"), + (leaf("x", "matches", "(a|aa)+"), "alternatives that start alike"), + (leaf("x", "matches", "(?:x|xy){2,}"), "alternatives that start alike"), + (leaf("x", "matches", r"(a)\1"), "refers back to a group"), + (leaf("x", "between", [5, 1]), "the first end is the lower one"), + (leaf("x", "between", [1, "2026-10-01"]), "two numbers or two times"), + (leaf("x", "between", ["2026-10-31", "2026-10-01"]), "the first end is the earlier one"), + (leaf("x", "between", [1]), "give the two ends"), + (leaf("x", "before", "tomorrow"), "compare with a time like"), + (leaf("x", "after", 5), "compare with a time like"), + (leaf("x", "within", "-2h"), "within can't point back"), + (leaf("x", "within", "0s"), "within is a time from now"), + (leaf("x", "within", "soon"), "within is a whole number"), + (leaf("x", "weekday", ["someday"]), "give the days"), + (leaf("x", "weekday", "sat"), "give the days"), + (leaf("x", "weekday", [8]), "give the days"), + (leaf("x", "time_between", ["22:00"]), "two times of day"), + (leaf("x", "time_between", ["25:00", "06:00"]), "two times of day"), + (leaf("x", "time_between", ["06:00", "6:00"]), "two different times"), + (leaf("x", "changed", "not a path"), "a path to keep them apart by"), + (leaf("x", "near", 1), "the test is one of exists, empty, equals"), + ] + for tree, said in bad: + with pytest.raises(ValueError, match=re.escape(said)): + match.check(tree) + with pytest.raises(ValueError, match="trigger mail's where, all.0: give a list"): + match.check({"all": [leaf("x", "in", "a")]}, "trigger mail's where") + # cleaned: days as three letters in week order, times of day with two digits, durations trimmed + assert match.check(leaf("x", "weekday", ["Sunday", 6, "mon"]))["value"] == ["mon", "sat", "sun"] + assert match.check(leaf("x", "time_between", ["6:00", "22:30"]))["value"] == ["06:00", "22:30"] + assert match.check(leaf("x", "within", " 2h "))["value"] == "2h" + assert match.check(leaf("x", "changed")) == {"path": "x", "op": "changed"} + assert match.check(leaf("x", "starts_with", 44))["value"] == "44" + for fine in ["trip|outing", "(trip|outing)+", "(cat|car)s?", r"\d+-\d+", r"(\w|-)+", "a{2,3}b+", "(ab)+c"]: + assert match.check(leaf("x", "matches", fine))["value"] == fine + + +def test_a_pattern_reads_a_long_value_only_so_far_and_gives_up_when_slow(monkeypatch): + long = "a" * 25_000 + " trip" + assert not passes("matches", "trip", long) # past MAX_INPUT characters + assert passes("matches", "trip", "a" * 19_000 + " trip") + monkeypatch.setattr(match, "PATTERN_SECONDS", 0.01) + slow = {"path": "x", "op": "matches", "value": "(a|aa)+$"} # never passes a check; here only to time it out + assert match.test_leaf(slow, {"x": "a" * 60 + "b"}) is False + + +def test_changed_compares_with_the_last_value_seen(): + tree = match.check(leaf("object.state", "changed")) + kept: dict = {} + + def seen(state): + return match.test(tree, {"object": {"state": state}}, {"state": kept}) + + assert seen("closed") is False # the first value seen is where it starts + assert seen("closed") is False + assert seen("open") is True + assert seen("open") is False + assert seen(None) is True + assert match.test(tree, {"object": {"state": "x"}}, None) is False # no state: nothing has changed + # each sensor its own: value names the path that keeps them apart + per = match.check(leaf("object.state", "changed", "actor.id")) + st: dict = {} + + def reading(sensor, state): + return match.test(per, {"actor": {"id": sensor}, "object": {"state": state}}, {"state": st}) + + assert [reading("s1", "on"), reading("s2", "off"), reading("s1", "on"), reading("s2", "on"), reading("s1", "off")] == [ + False, + False, + False, + True, + True, + ] + + +def test_one_state_answers_once_per_key_and_dry_keeps_nothing(db): + tree = match.check(leaf("object.state", "changed")) + a = {"object": {"state": "closed"}} + b = {"object": {"state": "open"}} + assert match.test(tree, a, {"state": match.State(db, 4, "door")}) is False + st = match.State(db, 4, "door") + assert match.test(tree, b, {"state": st}) is True + assert match.test(tree, b, {"state": st}) is True # the If and a node testing the same activity agree + assert match.test(tree, b, {"state": match.State(db, 4, "door")}) is False + assert match.test(tree, a, {"state": match.State(db, 4, "door", dry=True)}) is True # a dry run reads... + assert match.test(tree, a, {"state": match.State(db, 4, "door", dry=True)}) is True # ...and keeps nothing + assert match.test(tree, a, {"state": match.State(db, 4, "other")}) is False # each trigger its own + assert match.test(tree, a, {"state": match.State(db, 5, "door")}) is False # each automation its own + row = db.one("SELECT * FROM automation_state WHERE automation = 4 AND alias = 'door'") + assert row["kind"] == "last" and row["key"] == "object.state" and row["last"] == '"open"' and row["expires_at"] > row["at"] + + +def test_state_keeps_to_its_limit_and_is_tidied(db, monkeypatch): + monkeypatch.setattr(match, "MAX_OPEN", 3) + tree = match.check(leaf("v", "changed", "k")) + for k in range(5): + match.test(tree, {"k": k, "v": 1}, {"state": match.State(db, 7, "t")}) + assert db.one("SELECT count() AS n FROM automation_state WHERE automation = 7 GROUP ALL")["n"] == 3 + assert match.dropped(db, 7) == 2 + keys = sorted(r["key"] for r in db.rows("SELECT key FROM automation_state WHERE automation = 7")) + assert keys == ["v|2", "v|3", "v|4"] # the oldest went + db.q("UPDATE automation_state SET expires_at = '2000-01-01T00:00:00+00:00' WHERE key = 'v|2'") + assert match.tidy(db, now="2026-10-15T00:00:00+00:00") + assert len(db.rows("SELECT id FROM automation_state WHERE automation = 7")) == 2 + assert match.tidy(db) == 0 # once an hour, unless now is given + + +def test_the_added_tests_read_in_words(): + tree = { + "all": [ + leaf("object.from", "ends_with", "@stmarys.org"), + leaf("object.subject", "matches", "trip|outing"), + leaf("object.label", "in", ["school", "family"]), + {"not": leaf("object.label", "in", ["spam"])}, + leaf("object.size", "between", [1, 5]), + leaf("object.start", "within", "1d"), + leaf("object.start", "after", "now"), + leaf("published", "before", "-2h"), + leaf("published", "between", ["-1d", "2026-10-31"]), + leaf("now", "weekday", ["sat", "sun"]), + leaf("now", "time_between", ["22:00", "06:00"]), + leaf("object.code", "matches", r"^\d+$"), + leaf("object.state", "changed"), + {"not": leaf("object.state", "changed", "actor.id")}, + leaf("object.subject", "starts_with", "Re:"), + ] + } + assert match.describe(tree) == ( + "object.from ends with “@stmarys.org” and object.subject matches “trip” or “outing” and object.label is " + "“school” or “family” and object.label is not “spam” and object.size is between 1 and 5 and object.start is " + "within 1 day of now and object.start is after now and published is before 2 hours ago and published is " + "between 1 day ago and 2026-10-31 and now falls on a Saturday or Sunday and now is in the day between 22:00 " + "and 06:00 and object.code matches the pattern “^\\d+$” and object.state changed and object.state didn't " + "change for each actor.id and object.subject starts with “Re:”" + ) + assert match.describe(match.check(leaf("x", "not_in", ["a", "b", "c"]))) == "x is not “a”, “b” or “c”" + assert match.duration_words("1m") == "1 minute" and match.duration_words("-90s") == "90 seconds" diff --git a/fastapi_backend/uv.lock b/fastapi_backend/uv.lock index 1c1bdca8a..9842acede 100644 --- a/fastapi_backend/uv.lock +++ b/fastapi_backend/uv.lock @@ -1941,6 +1941,7 @@ dependencies = [ { name = "pypdf" }, { name = "pyyaml" }, { name = "rdflib" }, + { name = "regex" }, { name = "scipy" }, { name = "starlette" }, { name = "surrealdb" }, @@ -2089,6 +2090,7 @@ requires-dist = [ { name = "pyyaml", specifier = ">=6" }, { name = "rapidocr-onnxruntime", marker = "extra == 'rapidocr'" }, { name = "rdflib", specifier = ">=7.1,<8" }, + { name = "regex", specifier = ">=2024.11" }, { name = "scipy", specifier = ">=1.11" }, { name = "spacy", marker = "extra == 'spacy'", specifier = ">=3.8,<3.9" }, { name = "speechbrain", marker = "extra == 'voices'" },