From c69d580f20ea29e35f30923dd474c87b1ba1d648 Mon Sep 17 00:00:00 2001 From: Aditya Mukhopadhyay Date: Tue, 20 Oct 2026 20:26:55 +0530 Subject: [PATCH] docs(grammar): document line comments and Unicode identifiers (#650) Backport of #658 from v3. grammar.pest skips `;` line comments like whitespace, and symbols, variables, keywords and bind slots use ALPHABETIC and NUMBER, as the tokenizer does; `-` followed by a numeral is a number, not a symbol. The verifier's Comment and UnicodeLetter drift entries are removed. No parser change. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01PtCdzbJMSS1V8qEPaJ4jXz --- CHANGELOG.md | 3 ++- CLAUDE.md | 2 +- docs/TEST_COVERAGE.md | 2 +- src/query/datalog/parser/verifier.rs | 31 +++++++--------------------- tests/grammar/grammar.pest | 17 +++++++++------ 5 files changed, 22 insertions(+), 33 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 017cad27e1..2b235b2749 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -13,7 +13,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Tests -- **Parser vs grammar verifier (#643), backported from v3.0.0.** `src/query/datalog/parser/verifier.rs` checks, for every generated input, that the parser rejects what `tests/grammar/grammar.pest` rejects, and accepts what it accepts unless the error code is semantic (a listed code with a witness pest accepts). It runs every string of 1 to 3 characters over 102 characters (ASCII, `\t \n \r`, four non-ASCII stand-ins) through `parse_edn`, and every bracket-balanced sequence of up to `MINIGRAF_GRAMMAR_BOUND` (default 3) of 32 tokens in 15 command templates through `parse_datalog_command`, also with no separators up to 3 tokens. Deterministic; about 40 s by default in a debug build. On v2.x `INT-019` still mixes syntax and semantic parse errors (v3.0.0 splits it into `PRS-083`–`PRS-107`), so it counts as semantic here, and a syntax error coded `INT-019` that pest accepts is not caught (v3.0.0 checks that too). The drift it found is the same as on v3 and is listed in `KNOWN_DRIFT` by cause (comments and non-ASCII letters #650; empty sections, ignored keywords, rule and call heads, a lone `:` or `-`, unterminated strings #651; a fact of more than four elements, #649, is fixed above). `grammar.pest` now says what the parser means: a fact is exactly `[e a v]` plus an optional valid-time map and a retracted fact exactly `[e a v]`; `not`, `not-join`, `or`, `or-join` (and `and` inside an `or`) are never rule names; literal keywords end at a word boundary (`:find?x` is one keyword). A nightly workflow, `grammar-verifier.yml`, runs it at bound 4 on `main` and on `v3`, and opens or comments on an issue when it fails. +- **Parser vs grammar verifier (#643), backported from v3.0.0.** `src/query/datalog/parser/verifier.rs` checks, for every generated input, that the parser rejects what `tests/grammar/grammar.pest` rejects, and accepts what it accepts unless the error code is semantic (a listed code with a witness pest accepts). It runs every string of 1 to 3 characters over 102 characters (ASCII, `\t \n \r`, four non-ASCII stand-ins) through `parse_edn`, and every bracket-balanced sequence of up to `MINIGRAF_GRAMMAR_BOUND` (default 3) of 32 tokens in 15 command templates through `parse_datalog_command`, also with no separators up to 3 tokens. Deterministic; about 40 s by default in a debug build. On v2.x `INT-019` still mixes syntax and semantic parse errors (v3.0.0 splits it into `PRS-083`–`PRS-107`), so it counts as semantic here, and a syntax error coded `INT-019` that pest accepts is not caught (v3.0.0 checks that too). The drift it found is the same as on v3 and is listed in `KNOWN_DRIFT` by cause (comments and non-ASCII letters #650, since documented: see below; empty sections, ignored keywords, rule and call heads, a lone `:` or `-`, unterminated strings #651; a fact of more than four elements, #649, is fixed above). +- **The grammar documents comments and non-ASCII identifiers** (#650), backported from v3.0.0. The parser has always accepted both; `tests/grammar/grammar.pest` now says so. A `;` starts a comment that runs to the end of the line and is skipped like whitespace. Symbols, variables, keywords and bind slots take Unicode letters and numerals (`:person/名前`, `?ñ`), as Rust's `char::is_alphabetic` and `char::is_numeric` define them; a symbol starts with a letter or `_`, and `-` followed by a numeral starts a number, not a symbol. Nothing the parser accepts or rejects changes. `grammar.pest` now says what the parser means: a fact is exactly `[e a v]` plus an optional valid-time map and a retracted fact exactly `[e a v]`; `not`, `not-join`, `or`, `or-join` (and `and` inside an `or`) are never rule names; literal keywords end at a word boundary (`:find?x` is one keyword). A nightly workflow, `grammar-verifier.yml`, runs it at bound 4 on `main` and on `v3`, and opens or comments on an issue when it fails. ## v2.0.8 — 2026-10-20 diff --git a/CLAUDE.md b/CLAUDE.md index d3d53590a4..43cb5b56e9 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -118,7 +118,7 @@ cargo run < demos/demo_negation.txt 3. **`src/query/datalog/`** — Datalog engine - `parser.rs`: EDN/Datalog parser — `transact`, `retract`, `query`, `rule`, `:as-of`, `:valid-at`, `not`, `not-join` - - `parser/verifier.rs`: test-only (not wasm32) bounded-exhaustive check of the parser against `tests/grammar/grammar.pest` (#643): every 1–3 character string (`parse_edn`) and every balanced token filler up to `MINIGRAF_GRAMMAR_BOUND` (default 3) in 15 command templates (`parse_datalog_command`); pest rejects ⇒ parser rejects, pest accepts ⇒ parser accepts or raises a `SEMANTIC` code (`INT-019` counts as semantic on v2.x); today's drift is `KNOWN_DRIFT`, one entry per cause, removed by its fix (#650, #651); `grammar-verifier.yml` runs bound 4 nightly + - `parser/verifier.rs`: test-only (not wasm32) bounded-exhaustive check of the parser against `tests/grammar/grammar.pest` (#643): every 1–3 character string (`parse_edn`) and every balanced token filler up to `MINIGRAF_GRAMMAR_BOUND` (default 3) in 15 command templates (`parse_datalog_command`); pest rejects ⇒ parser rejects, pest accepts ⇒ parser accepts or raises a `SEMANTIC` code (`INT-019` counts as semantic on v2.x); today's drift is `KNOWN_DRIFT`, one entry per cause, removed by its fix (#651; #650 fixed); `grammar-verifier.yml` runs bound 4 nightly - `executor.rs`: Query executor — temporal filter (tx-time → net-assert → valid-time), not/not-join post-filter - `matcher.rs`: Pattern matching with variable unification; `edn_to_value`, `edn_to_entity_id` - `evaluator.rs`: `RecursiveEvaluator` (semi-naive), `StratifiedEvaluator` (repeats a stratum whose rules recurse through a `not`/`or` rule until it derives nothing new, #633), `evaluate_not_join` diff --git a/docs/TEST_COVERAGE.md b/docs/TEST_COVERAGE.md index 1b3e18f851..4fdc50e500 100644 --- a/docs/TEST_COVERAGE.md +++ b/docs/TEST_COVERAGE.md @@ -22,7 +22,7 @@ For v2.0.5, torn-WAL-tail tests (#513) check that opening a WAL writer cuts off `tests/error_codes_prs_test.rs` checks that a fact of five or six elements, with or without a map 4th, fails with `PRS-108`, and that a transaction holding one stores none of its facts (#649). -Since v2.0.8, `src/query/datalog/parser/verifier.rs` (4 tests, backported from v3.0.0; #643) checks the parser against `tests/grammar/grammar.pest` on about 1.07M lexical strings and 555K structural inputs at the default bound (7.6M at `MINIGRAF_GRAMMAR_BOUND=4`, run nightly by `grammar-verifier.yml`): pest rejects ⇒ the parser rejects, pest accepts ⇒ the parser accepts or raises a `SEMANTIC` code with a witness (`INT-019` counts as semantic on v2.x). Known drift is listed by cause in `KNOWN_DRIFT` (#650, #651). `tests/grammar_conformance.rs` now classes a fact with too few elements and a retract of the wrong arity as syntax errors, since `grammar.pest`'s fact is exact. +Since v2.0.8, `src/query/datalog/parser/verifier.rs` (4 tests, backported from v3.0.0; #643) checks the parser against `tests/grammar/grammar.pest` on about 1.07M lexical strings and 555K structural inputs at the default bound (7.6M at `MINIGRAF_GRAMMAR_BOUND=4`, run nightly by `grammar-verifier.yml`): pest rejects ⇒ the parser rejects, pest accepts ⇒ the parser accepts or raises a `SEMANTIC` code with a witness (`INT-019` counts as semantic on v2.x). Known drift is listed by cause in `KNOWN_DRIFT` (#651). `tests/grammar_conformance.rs` now classes a fact with too few elements and a retract of the wrong arity as syntax errors, since `grammar.pest`'s fact is exact. Not counted above: the `BrowserDb` tests run only on wasm32 with `wasm-bindgen-test` in a browser. Two of them check that the IndexedDB flush returns every page a write dirtied, and that a dirty page that cannot be read is an error rather than left out. diff --git a/src/query/datalog/parser/verifier.rs b/src/query/datalog/parser/verifier.rs index 4cebcb7c53..560f7e9678 100644 --- a/src/query/datalog/parser/verifier.rs +++ b/src/query/datalog/parser/verifier.rs @@ -221,12 +221,8 @@ fn is_semantic(code: &str) -> bool { #[derive(Clone, Copy, PartialEq, Eq, Debug)] enum Cause { // Lexical. - /// `;` starts a line comment in the parser; the grammar has no comments. - Comment, /// A string with no closing quote reads to the end of the input. UnterminatedString, - /// A non-ASCII letter or digit in a symbol or keyword (the grammar is ASCII). - UnicodeLetter, /// `:` with no keyword character after it is the keyword `:`. BareColon, /// `-` with no word character after it is the symbol `-`. @@ -257,11 +253,7 @@ enum Cause { impl Cause { fn level(self) -> Level { match self { - Cause::Comment - | Cause::UnterminatedString - | Cause::UnicodeLetter - | Cause::BareColon - | Cause::LoneMinus => Level::Lexical, + Cause::UnterminatedString | Cause::BareColon | Cause::LoneMinus => Level::Lexical, _ => Level::Structural, } } @@ -275,21 +267,11 @@ struct Drift { } const KNOWN_DRIFT: &[Drift] = &[ - Drift { - cause: Cause::Comment, - issue: 650, - min_k: 0, - }, Drift { cause: Cause::UnterminatedString, issue: 651, min_k: 0, }, - Drift { - cause: Cause::UnicodeLetter, - issue: 650, - min_k: 0, - }, Drift { cause: Cause::BareColon, issue: 651, @@ -370,14 +352,12 @@ fn lexical_cause(text: &str) -> Option { } } } - ';' => return Some(Cause::Comment), ':' if !next.is_some_and(symbol_char) => return Some(Cause::BareColon), '-' if !next.is_some_and(symbol_char) && !(i > 0 && (symbol_char(c[i - 1]) || matches!(c[i - 1], ':' | '#' | '$'))) => { return Some(Cause::LoneMinus); } - ch if !ch.is_ascii() && ch.is_alphanumeric() => return Some(Cause::UnicodeLetter), _ => {} } i += 1; @@ -455,11 +435,14 @@ fn query_cause(items: &[EdnValue]) -> Option { /// Whether pest's `plain_sym` reads all of `s`. fn is_plain_symbol(s: &str) -> bool { - let word = |c: char| c.is_ascii_alphanumeric() || matches!(c, '?' | '_' | '-' | '/'); let mut chars = s.chars(); match chars.next() { - Some(c) if c.is_ascii_alphabetic() || c == '_' => chars.all(word), - Some('-') => !chars.as_str().is_empty() && chars.all(word), + Some(c) if c.is_alphabetic() || c == '_' => chars.all(symbol_char), + Some('-') => { + !chars.as_str().is_empty() + && !chars.as_str().starts_with(char::is_numeric) + && chars.all(symbol_char) + } _ => false, } } diff --git a/tests/grammar/grammar.pest b/tests/grammar/grammar.pest index 5fae333503..b3793f9faa 100644 --- a/tests/grammar/grammar.pest +++ b/tests/grammar/grammar.pest @@ -1,9 +1,13 @@ // ── WHITESPACE (auto-skipped between every token) ───────────────────────── WHITESPACE = _{ " " | "\t" | "\n" | "\r" | "," } +// Line comment: ";" to the end of the line, skipped like whitespace (#650). +COMMENT = _{ ";" ~ (!"\n" ~ ANY)* } + // ── WORD-BOUNDARY HELPER ────────────────────────────────────────────────── -// Used in negative lookaheads to enforce keyword boundaries. -word_cont = _{ ASCII_ALPHANUMERIC | "?" | "_" | "-" | "/" } +// Used in negative lookaheads to enforce keyword boundaries. Letters and numerals are +// Unicode's (Alphabetic; Nd, Nl, No), as the parser reads them (#650). +word_cont = _{ ALPHABETIC | NUMBER | "?" | "_" | "-" | "/" } // Literal keywords: atomic, so that no whitespace is skipped before the // boundary check (":find?x" is one keyword, as the parser reads it). @@ -52,7 +56,7 @@ string_lit = @{ "\"" ~ str_char* ~ "\"" } uuid_lit = { "#uuid" ~ string_lit } // Prepared-query bind slot: $name -bind_slot = @{ "$" ~ (ASCII_ALPHANUMERIC | "_" | "-")+ } +bind_slot = @{ "$" ~ (ALPHABETIC | NUMBER | "_" | "-")+ } // Keyword: starts with ":", followed by word constituents (incl. "?") keyword = @{ ":" ~ word_cont+ } @@ -63,11 +67,12 @@ variable = @{ "?" ~ word_cont* } // Operator symbols used in expressions op_sym = @{ "<=" | ">=" | "!=" | "<" | ">" | "=" | "+" | "*" | "/" } -// Plain symbol: letter or underscore start; or "-" followed by at least one word char. +// Plain symbol: letter or underscore start; or "-" followed by at least one word char +// that is not a numeral (after "-" the parser reads a numeral as a number). // Negative lookaheads prevent "true", "false", "nil" from matching as plain symbols. plain_sym = @{ - !boolean_lit ~ !nil_lit ~ (ASCII_ALPHA | "_") ~ word_cont* - | "-" ~ word_cont+ + !boolean_lit ~ !nil_lit ~ (ALPHABETIC | "_") ~ word_cont* + | "-" ~ !NUMBER ~ word_cont+ } // ── EDN VALUE (any EDN atom or container) ─────────────────────────────────