From 8a83fe74ce69eddb888d6e1800393ddc6216de86 Mon Sep 17 00:00:00 2001 From: Adeel Ahmad Date: Sat, 17 Oct 2026 02:08:35 +0000 Subject: [PATCH] feat(analysis): fix transcripts' grammar with a LanguageTool server With transcripts cleaned up and Settings > Analysis > LanguageTool server set (none by default), the clean step sends a transcript's lines to that server, a batch at a time, and makes its grammar and punctuation fixes ("he go" > "he goes", "a apple" > "an apple"). Spelling, style and wording suggestions are left out, and so is any fix to a word with a capital, so names stay as said. Offsets are read in UTF-16 units, as LanguageTool counts them. No redirect is followed, each call is an activity ledger row, and when the server doesn't answer the rules' changes stand and the job log says so. The compose files gain an opt-in `languagetool` profile (erikvl87/languagetool 6.8, on Lens's network only). language_tool_python, which the pasted script used, is GPL-3.0 and starts Java itself, so Lens talks to a server instead. Docs, changelog, dev log and threat model updated. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_014z3r72SB2quWGKbgGt46mZ --- DEVLOG.md | 6 + README.md | 4 +- changelog.d/521.md | 5 + docker-compose.prod.yml | 13 ++ docker-compose.yml | 12 ++ docs/processing.md | 12 +- docs/requirements.md | 1 + fastapi_backend/app/domain/cleanup.py | 136 ++++++++++++++++-- fastapi_backend/app/domain/settings.py | 9 ++ fastapi_backend/app/domain/store.py | 6 +- fastapi_backend/tests/domain/test_cleanup.py | 98 +++++++++++++ .../components/settings/bodies.tsx | 1 + nextjs-frontend/components/settings/model.ts | 10 ++ threat-model.json | 6 +- threat-model.md | 4 +- threat-model.yaml | 4 +- 16 files changed, 306 insertions(+), 21 deletions(-) create mode 100644 changelog.d/521.md diff --git a/DEVLOG.md b/DEVLOG.md index ea7965f35..eed328fcb 100644 --- a/DEVLOG.md +++ b/DEVLOG.md @@ -63,6 +63,12 @@ from a LanguageTool server, spaCy fetched for named entities, and dates found an Refine later: revert a whole cleanup at once; cleanup rules for other languages' fillers; per-namespace switch. +Grammar: the clean step takes a LanguageTool server's GRAMMAR, PUNCTUATION and TYPOGRAPHY fixes, and EN_A_VS_AN (filed +under MISC) (analysis.cleanup_grammar_url, the compose files' `languagetool` profile), never spelling, style or a word +with a capital. language_tool_python from the script is GPL-3.0 and starts Java itself, so Lens talks to a server +instead. Refine later: start the profile from Settings through the machine agent; per-namespace grammar; LanguageTool's +other categories as opt-in switches. + ## 2026-10-17 · Describing pictures with Moondream on this machine Moondream is the third describe engine (`llm.describe_engine` `moondream`), opt-in, through Hugging Face transformers diff --git a/README.md b/README.md index a5eb642ca..7aa9ecde3 100644 --- a/README.md +++ b/README.md @@ -131,8 +131,8 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p asset, a topic or an entity, in a Similar tab on every asset or by selecting text anywhere, and, with anytopdf, photos by what they show. Pages and shots are described in words by the model provider's vision model or, off by default, by Florence-2 or Moondream on your own server. Transcripts can be cleaned up before analysis (stutters, "um"s, spacing - and, with a local model, punctuation put back), entities found by spaCy instead of the rules, and the dates people say - read as days on the calendar, each off by default. + and, with a local model, punctuation put back, and grammar by a LanguageTool server you run), entities found by spaCy + instead of the rules, and the dates people say read as days on the calendar, each off by default. ([Graph](docs/graph.md), [entity pages](docs/notes.md#entity-pages), [similar content](docs/processing.md#similar-content), [graph history](docs/graph-history.md), [topics](docs/topics.md), [video](docs/video.md), [NSFW classification](docs/nsfw.md), diff --git a/changelog.d/521.md b/changelog.d/521.md new file mode 100644 index 000000000..b117c8754 --- /dev/null +++ b/changelog.d/521.md @@ -0,0 +1,5 @@ +- **Grammar fixed by LanguageTool, when you run one.** With transcripts cleaned up and Settings → Analysis → + LanguageTool server set (none by default; the compose files' new `languagetool` profile starts one at + `http://languagetool:8010`), the clean step makes its grammar and punctuation fixes too, leaving spelling, style and + names alone. Each change is in Change history with Revert; without the server the rules run alone + ([Cleaning up transcripts](docs/processing.md#cleaning-up-transcripts)). diff --git a/docker-compose.prod.yml b/docker-compose.prod.yml index a749c5342..240d0fa58 100644 --- a/docker-compose.prod.yml +++ b/docker-compose.prod.yml @@ -202,6 +202,19 @@ services: restart: unless-stopped networks: [lens] + # LanguageTool (docs/processing.md#cleaning-up-transcripts): grammar and punctuation fixes for the clean step. Opt-in, + # about 1 GB of memory: `docker compose --profile languagetool up -d` (or COMPOSE_PROFILES=languagetool in .env), then + # Settings → Analysis → LanguageTool server: http://languagetool:8010. Reachable only inside Lens's network. + languagetool: + image: erikvl87/languagetool:${LANGUAGETOOL_VERSION:-6.8} + profiles: [languagetool] + security_opt: ["no-new-privileges:true"] + environment: + Java_Xms: 256m + Java_Xmx: ${LANGUAGETOOL_HEAP:-768m} + restart: unless-stopped + networks: [lens] + # The MCP runner (mcp-runner/runner.py, docs/assistant.md): MCP servers that run as a program (stdio, usually npx or # uvx), which admins add as extensions. Opt-in: `docker compose --profile mcp up -d`, or COMPOSE_PROFILES=mcp in .env. # The sandbox: none of Lens's data or keys, no capabilities, a read-only image, and a network shared only with the diff --git a/docker-compose.yml b/docker-compose.yml index 7f1ed18f6..9968eae71 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -173,6 +173,18 @@ services: JAVA_TOOL_OPTIONS: -Xmx${TIKA_HEAP:-512m} networks: [lens] + # LanguageTool (docs/processing.md#cleaning-up-transcripts): grammar and punctuation fixes for the clean step. Opt-in, + # about 1 GB of memory: `docker compose --profile languagetool up -d` (or COMPOSE_PROFILES=languagetool in .env), then + # Settings → Analysis → LanguageTool server: http://languagetool:8010. Reachable only inside Lens's network. + languagetool: + image: erikvl87/languagetool:${LANGUAGETOOL_VERSION:-6.8} + profiles: [languagetool] + security_opt: ["no-new-privileges:true"] + environment: + Java_Xms: 256m + Java_Xmx: ${LANGUAGETOOL_HEAP:-768m} + networks: [lens] + # The MCP runner (mcp-runner/runner.py, docs/assistant.md): MCP servers that run as a program (stdio, usually npx or # uvx), which admins add as extensions. Opt-in: `docker compose --profile mcp up -d`, or COMPOSE_PROFILES=mcp in .env. # The sandbox: none of Lens's data or keys, no capabilities, a read-only image, and a network shared only with the diff --git a/docs/processing.md b/docs/processing.md index ab26f2861..d3b60edd3 100644 --- a/docs/processing.md +++ b/docs/processing.md @@ -27,8 +27,8 @@ Every step takes `--ns`, `--limit` and `--force`; `run` does them all, and a loc Speech comes out of transcription as it was said, and the extractor, chapters, keywords and search all read it. With `analysis.cleanup` on (Settings → Analysis → Clean up transcripts before analysis; off by default), the `clean` step, -which the standard pipeline runs just before `analyze`, tidies each line of a transcript with rules. It needs no model, -fetches nothing and sends nothing out: +which the standard pipeline runs just before `analyze`, tidies each line of a transcript with rules. The rules need no +model, fetch nothing and send nothing out: - a word said three times or more in a row is kept once ("I I I looked" → "I looked"), and a doubled pronoun, article or conjunction ("the the", "I, I") once; "had had", "that that" and "very very" stay; @@ -44,6 +44,14 @@ fetches nothing and sends nothing out: each line marks once. The model, Transformers and PyTorch (about 2 GB) are fetched the first time a clean step needs them ([Components](components.md)); until then, and in other languages, the rules run alone and the job's log says why. +- grammar (`analysis.cleanup_grammar_url`, LanguageTool server; none): a [LanguageTool](https://languagetool.org) + server's grammar and punctuation fixes are made too ("he go" → "he goes", "a apple" → "an apple"), and only those: + spelling, style, wording and repeated-word ("very very") suggestions are left out, and so is any fix to a word + with a capital in it, so names stay as they were said. Start one with the compose files' `languagetool` profile + (`docker compose --profile languagetool up -d`, about 1 GB of memory) and set `http://languagetool:8010`. The + transcript's lines go to that address and nowhere else (no redirect is followed), so a server outside your network + sees them; each call is in the activity ledger. When it doesn't answer, the rules' changes stand and the job's log + says so. These are English rules ("um" is a word in German and Portuguese), so they apply to English lines: the line's or the asset's language when transcription gave one, else when the text reads as English. In other languages only a word diff --git a/docs/requirements.md b/docs/requirements.md index a145a060d..caf745afa 100644 --- a/docs/requirements.md +++ b/docs/requirements.md @@ -160,6 +160,7 @@ request, to another check that proves the same thing; loosening a requirement or | Or by Moondream on this server, off by default, fetched at a pinned revision only where the chosen model can run, and saying why where it can't | [test_moondream.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_moondream.py) `test_moondream_describes_pictures_without_a_model_provider`, `test_where_each_moondream_can_run`, `test_on_a_cpu_moondream_3_is_never_fetched_and_the_step_says_why`, `test_moondream_is_fetched_pinned`, `test_admins_and_agents_choose_moondream` | | Transcripts cleaned up before analysis (stutters, "um"s, spacing), off by default, each change in the edit history with Revert and people's corrections left alone | [test_cleanup.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_cleanup.py) `test_off_by_default_changes_nothing`, `test_clean_step_tidies_lines_and_keeps_them_in_the_history`, `test_people_s_corrections_outrank_the_rules` | | Punctuation put back in transcripts that lack it by a local model, fetched only when picked | [test_cleanup.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_cleanup.py) `test_punctuation_put_back_where_it_is_missing`, `test_without_the_model_the_rules_run_alone`, `test_the_model_is_fetched_only_when_it_is_asked_for` | +| Grammar fixed by a LanguageTool server you run, when its address is set: grammar fixes only, never spelling or a word with a capital | [test_cleanup.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_cleanup.py) `test_grammar_fixes_from_languagetool_leave_names_and_spelling`, `test_without_its_server_the_rules_run_alone`, `test_the_grammar_server_is_an_address_without_a_password` | | Entities found by spaCy, fetched the first time it's picked, with the defined entities first and the rules until it's there | [test_spacy_entities.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_spacy_entities.py) `test_defined_entities_come_first_then_spacy_s_names`, `test_the_analyze_step_and_workflows_use_it`, `test_without_spacy_the_rules_find_them`, `test_spacy_is_fetched_only_when_it_is_picked` | | Dates said in an asset read as days ("next Tuesday", "last week"), off by default, with dates that read two ways left out | [test_dates.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_dates.py) `test_dates_are_read_as_days`, `test_what_could_be_two_dates_is_left_out`, `test_off_by_default_reads_none` | diff --git a/fastapi_backend/app/domain/cleanup.py b/fastapi_backend/app/domain/cleanup.py index 763273ea7..8f50f98e2 100644 --- a/fastapi_backend/app/domain/cleanup.py +++ b/fastapi_backend/app/domain/cleanup.py @@ -22,6 +22,12 @@ once. The model and PyTorch (about 2 GB) are fetched the first time a clean step needs them (components.py); until then, or where it can't run, the rules run alone. +Grammar (`analysis.cleanup_grammar_url`, none by default): with a LanguageTool server's address (the compose files' +`languagetool` profile: http://languagetool:8010), its grammar and punctuation fixes are made too ("he go" → "he goes", +"a apple" → "an apple"), and only those: spelling, style, wording and repeated-word suggestions are left out, and so is +any fix to a word with a capital in it, so names stay as they were said. A line goes to that server and nowhere else; when it +doesn't answer, the rules' changes stand alone. + Every line it changes is kept in the asset's edit history, by "text cleanup": Change history lists them and Revert puts a line back. A line someone corrected is never cleaned, since people's corrections outrank the rules, and running the rules again on what they made changes nothing. @@ -29,9 +35,13 @@ from __future__ import annotations +import json import re +import urllib.error +import urllib.parse +import urllib.request -from . import store +from . import activity, store R = store.R CLEANER = "text cleanup" # who the edit history says changed a line @@ -258,6 +268,104 @@ def _punctuation(cfg, rec, segs): return texts, [segs[k]["id"] for k in todo], f"punctuation restored by {PUNCTUATION_MODEL[0]}" +# ---------- grammar, by a LanguageTool server ---------- +# LanguageTool's fixes taken: its grammar, punctuation and typography categories, and "a" or "an", which it files under +# MISC. Spelling (TYPOS), style, wording and its repeated-word rule ("very very") would rewrite how people talk. +GRAMMAR = {"GRAMMAR", "PUNCTUATION", "TYPOGRAPHY"} +GRAMMAR_RULES = {"EN_A_VS_AN"} +GRAMMAR_CHARS = 10_000 # text sent to the server at once +GRAMMAR_ANSWER = 4 << 20 # the most of an answer read +GRAMMAR_SECONDS = 60 + + +def _check(url, text, lang, cfg): + """LanguageTool's matches for `text` ([{offset, length, replacements, rule}]); Unavailable when it doesn't answer.""" + body = urllib.parse.urlencode({"text": text, "language": lang or "auto"}).encode() + req = urllib.request.Request(f"{url}/v2/check", data=body, headers={"Accept": "application/json"}, method="POST") + host = urllib.parse.urlsplit(url).hostname + try: + with activity.call("grammar.check", cfg, detail={"host": host, "chars": len(text)}): + with _opener.open(req, timeout=GRAMMAR_SECONDS) as r: + got = json.loads(r.read(GRAMMAR_ANSWER)) + except (OSError, ValueError) as e: + raise Unavailable(f"the LanguageTool server at {host} didn't answer ({e})") from None + return got.get("matches") or [] if isinstance(got, dict) else [] + + +class _NoRedirect(urllib.request.HTTPRedirectHandler): + def redirect_request(self, *args, **kwargs): # the text goes to the address set, and nowhere it sends it on + return None + + +_opener = urllib.request.build_opener(_NoRedirect) + + +def _u16(text): + return len(text.encode("utf-16-le")) // 2 + + +def _cp(text, units): + """A UTF-16 offset into `text` (LanguageTool's, from Java) as Python's.""" + if text.isascii(): + return units + n = 0 + for i, ch in enumerate(text): + if n >= units: + return i + n += 2 if ord(ch) > 0xFFFF else 1 + return len(text) + + +def _fix(text, matches): + """`text` with the fixes taken (GRAMMAR, GRAMMAR_RULES): the first replacement of each, none to a word with a + capital, and none that overlaps another.""" + upto = len(text) + for m in sorted(matches, key=lambda m: -m["offset"]): + a, b = _cp(text, m["offset"]), _cp(text, m["offset"] + m["length"]) + rule = m.get("rule") or {} + cat = ((rule.get("category") or {}).get("id") or "").upper() + reps = [r.get("value") for r in m.get("replacements") or [] if isinstance(r.get("value"), str)] + taken = cat in GRAMMAR or rule.get("id") in GRAMMAR_RULES + if not taken or not reps or a < 0 or b > upto or any(c.isupper() for c in text[a:b] + reps[0]): + continue + text, upto = text[:a] + reps[0] + text[b:], a + return text + + +def grammar(cfg, texts, lang=None): + """The texts with a LanguageTool server's grammar and punctuation fixes (analysis.cleanup_grammar_url), sent a + batch of lines at a time. Unavailable when no server is set or it doesn't answer.""" + url = ((cfg.get("analysis") or {}).get("cleanup_grammar_url") or "").rstrip("/") + if not url: + raise Unavailable("no LanguageTool server is set") + batches, batch, size = [], [], 0 + for k, t in enumerate(texts): + if not t.strip(): + continue + if batch and size + len(t) + 2 > GRAMMAR_CHARS: + batches.append(batch) + batch, size = [], 0 + batch.append(k) + size += len(t) + 2 + batches += [batch] if batch else [] + lang = lang if lang and lang not in ("auto", "nospeech") else "auto" + out = list(texts) + for batch in batches: # a batch's lines one after another, a blank line between them + joined, starts, at = "", [], 0 + for j in batch: + starts.append(at) # where each line starts, in UTF-16 units as LanguageTool counts + joined += texts[j] + "\n\n" + at += _u16(texts[j]) + 2 + found = _check(url, joined, lang, cfg) + for j, at in zip(batch, starts, strict=True): + end = at + _u16(texts[j]) + mine = [ + {**m, "offset": m["offset"] - at} for m in found if at <= m.get("offset", -1) and m["offset"] + m.get("length", 0) <= end + ] + out[j] = _fix(texts[j], mine) + return out + + # ---------- an asset's lines ---------- def plan(db, cfg, rid): """What cleaning an asset does: {changes: [(segment id, idx, text, cleaned)], punctuated: [segment ids], note}. @@ -272,20 +380,32 @@ def plan(db, cfg, rid): fillers = bool((cfg.get("analysis") or {}).get("cleanup_fillers", True)) whole = english(rec.get("language"), [s.get("text") or "" for s in segs[:400]]) marked, punctuated, note = _punctuation(cfg, rec, segs) - out, prev = [], None - for s, text in zip(segs, marked): + cleaned, prev = [], None + for s, text in zip(segs, marked, strict=True): ended = prev is None or prev["speaker"] != s.get("speaker") or SENTENCE_END.search(prev["text"]) is not None prev = {"speaker": s.get("speaker"), "text": text} if s["idx"] in people or not text.strip(): + cleaned.append(None) continue lang = s.get("lang") en = english(lang, [text]) if lang and lang not in ("auto", "nospeech") else whole - new = clean_line(text, en, fillers, ended) - if new and new != (s.get("text") or ""): - out.append((s["id"], s["idx"], s.get("text") or "", new)) - prev["text"] = new + prev["text"] = clean_line(text, en, fillers, ended) or text + cleaned.append(prev["text"]) + notes = [note] if note else [] + if (cfg.get("analysis") or {}).get("cleanup_grammar_url"): + todo = [k for k, t in enumerate(cleaned) if t] + try: + fixed = grammar(cfg, [cleaned[k] for k in todo], rec.get("language")) + for k, t in zip(todo, fixed, strict=True): + cleaned[k] = t + notes.append("grammar checked by LanguageTool") + except Unavailable as e: + notes.append(f"grammar not checked: {e}") + out = [ + (s["id"], s["idx"], s.get("text") or "", new) for s, new in zip(segs, cleaned, strict=True) if new and new != (s.get("text") or "") + ] kept = {s["id"] for s in segs if s["idx"] in people} - return {"changes": out, "punctuated": [i for i in punctuated if i not in kept], "note": note} + return {"changes": out, "punctuated": [i for i in punctuated if i not in kept], "note": "; ".join(notes) or None} def clean_recording(db, cfg, rid, say=None): diff --git a/fastapi_backend/app/domain/settings.py b/fastapi_backend/app/domain/settings.py index b0ffeb5e9..d44103899 100644 --- a/fastapi_backend/app/domain/settings.py +++ b/fastapi_backend/app/domain/settings.py @@ -632,6 +632,15 @@ def _check(section, key, value, default): if not (isinstance(value, str) and VIEWER_URL.match(value.strip())): raise ValueError("documents.anytopdf_url is a conversion node's http(s) address, such as https://convert.home:8640") return value.strip().rstrip("/") + if (section, key) == ("analysis", "cleanup_grammar_url"): + if value in (None, ""): + return None + u = urllib.parse.urlsplit(value.strip()) if isinstance(value, str) else None + if not (u and VIEWER_URL.match(value.strip()) and u.hostname and u.username is None and u.password is None): + raise ValueError( + "analysis.cleanup_grammar_url is a LanguageTool server's http(s) address with no password in it, such as http://languagetool:8010" + ) + return value.strip().rstrip("/") if (section, key) == ("documents", "tika_url"): if value in (None, ""): return None diff --git a/fastapi_backend/app/domain/store.py b/fastapi_backend/app/domain/store.py index 028e3b017..be6c2f503 100644 --- a/fastapi_backend/app/domain/store.py +++ b/fastapi_backend/app/domain/store.py @@ -156,8 +156,9 @@ def labels(): # anytopdf: the places and dates anytopdf (run here) finds, as hints for the extractor (enrich.py); dates: the dates # each line gives kept as days on the calendar (dates.py); cleanup: the # clean step tidies transcripts before analysis (cleanup.py), cleanup_fillers taking out "um" and "uh" too, and - # cleanup_punctuation (off, missing, always) putting marks back with a local model; typing: which model types the - # names the rules leave terms (name_types.asking) + # cleanup_punctuation (off, missing, always) putting marks back with a local model, cleanup_grammar_url a + # LanguageTool server's grammar and punctuation fixes (the compose files' `languagetool` profile); typing: which + # model types the names the rules leave terms (name_types.asking) "analysis": { "entities": "rules", "typing": "auto", @@ -168,6 +169,7 @@ def labels(): "cleanup": False, "cleanup_fillers": True, "cleanup_punctuation": "off", + "cleanup_grammar_url": None, }, # chat_models: the models people may pick in Chat; empty: whatever the model server lists (docs/configuration.md) "llm": { diff --git a/fastapi_backend/tests/domain/test_cleanup.py b/fastapi_backend/tests/domain/test_cleanup.py index ad2b474a9..f3230e403 100644 --- a/fastapi_backend/tests/domain/test_cleanup.py +++ b/fastapi_backend/tests/domain/test_cleanup.py @@ -3,6 +3,9 @@ from __future__ import annotations +import json +import re + import pytest from app.domain import cleanup, ingest, jobs, store @@ -228,3 +231,98 @@ def test_the_model_is_fetched_only_when_it_is_asked_for(cfg): assert not c.needed(cfg, m) # cleanup itself is off cfg["analysis"]["cleanup"] = True assert c.needed(cfg, m) and c.serves(cfg, m) == {"clean"} + + +# ---------- grammar ---------- +# what LanguageTool 6.8 answers for these: the rule, its category, and the words it matches +FIXES = [ + (r"(?<=he )go\b", "goes", "HE_VERB_AGR", "GRAMMAR"), + (r"\ba(?= apple)", "an", "EN_A_VS_AN", "MISC"), + (r"\brealy\b", "really", "MORFOLOGIK_RULE_EN_US", "TYPOS"), + (r"very very", "very", "ENGLISH_WORD_REPEAT_RULE", "MISC"), + (r"Dyno", "dyno", "MADE_UP", "GRAMMAR"), +] +WORDY = """Ann|N|so he go to the shop 🙂 he go home +Ben|N|I ate a apple and it was realy very very good +Ann|N|we met Dyno Therapeutics""" + + +@pytest.fixture +def languagetool(): + """A LanguageTool server that knows FIXES, counting offsets in UTF-16 units as the real one (in Java) does.""" + import http.server + import threading + import urllib.parse + + seen = [] + + class Handler(http.server.BaseHTTPRequestHandler): + def do_POST(self): # noqa: N802 + form = urllib.parse.parse_qs(self.rfile.read(int(self.headers["Content-Length"])).decode()) + text = form["text"][0] + seen.append((form["language"][0], text)) + u16 = lambda s: len(s.encode("utf-16-le")) // 2 # noqa: E731 + matches = [ + { + "offset": u16(text[: m.start()]), + "length": u16(m.group()), + "replacements": [{"value": right}], + "rule": {"id": rule, "category": {"id": cat}}, + } + for wrong, right, rule, cat in FIXES + for m in re.finditer(wrong, text) + ] + body = json.dumps({"matches": matches}).encode() + self.send_response(200) + self.send_header("Content-Type", "application/json") + self.send_header("Content-Length", str(len(body))) + self.end_headers() + self.wfile.write(body) + + def log_message(self, *args): + pass + + srv = http.server.ThreadingHTTPServer(("127.0.0.1", 0), Handler) + threading.Thread(target=srv.serve_forever, daemon=True).start() + yield f"http://127.0.0.1:{srv.server_port}", seen + srv.shutdown() + + +def test_grammar_fixes_from_languagetool_leave_names_and_spelling(db, cfg, languagetool): + url, seen = languagetool + cfg["analysis"].update(cleanup=True, cleanup_grammar_url=url) + rid = ingest.import_text(db, cfg, "pods", WORDY, "Wordy", "auto") + jid = jobs.enqueue(db, rid, ["clean"]) + drain(db, cfg) + assert _lines(db, rid) == [ + "So he goes to the shop 🙂 he goes home", # after the emoji too: offsets are counted as LanguageTool counts them + "I ate an apple and it was realy very very good", # nor spelling, nor how people talk + "We met Dyno Therapeutics", # a word with a capital isn't changed + ] + assert len(seen) == 1 and seen[0][0] == "auto" # the lines go at once, a blank line between them + log = db.one("SELECT log FROM $j", j=R("job", jid))["log"] + assert any("grammar checked by LanguageTool" in line for line in log) + edits = db.rows("SELECT by FROM segment_edit WHERE recording = $r", r=rid) + assert [e["by"] for e in edits] == [cleanup.CLEANER] * 3 + + +def test_without_its_server_the_rules_run_alone(db, cfg): + cfg["analysis"].update(cleanup=True, cleanup_grammar_url="http://127.0.0.1:9") + rid = ingest.import_text(db, cfg, "pods", WORDY, "Wordy", "auto") + jid = jobs.enqueue(db, rid, ["clean"]) + drain(db, cfg) + assert _lines(db, rid)[0] == "So he go to the shop 🙂 he go home" + log = db.one("SELECT log FROM $j", j=R("job", jid))["log"] + assert any("grammar not checked: the LanguageTool server at 127.0.0.1 didn't answer" in line for line in log) + + +def test_the_grammar_server_is_an_address_without_a_password(): + from app.domain import settings + + def check(v): + return settings._check("analysis", "cleanup_grammar_url", v, None) + + assert check(" http://languagetool:8010/ ") == "http://languagetool:8010" and check("") is None + for bad in ("ftp://lt", "http://user:pw@lt:8010", "languagetool"): + with pytest.raises(ValueError, match="LanguageTool server"): + check(bad) diff --git a/nextjs-frontend/components/settings/bodies.tsx b/nextjs-frontend/components/settings/bodies.tsx index 029ca5715..44c618639 100644 --- a/nextjs-frontend/components/settings/bodies.tsx +++ b/nextjs-frontend/components/settings/bodies.tsx @@ -254,6 +254,7 @@ export function SectionBody({ ctx }: { ctx: BodyCtx }) {
+
)} diff --git a/nextjs-frontend/components/settings/model.ts b/nextjs-frontend/components/settings/model.ts index 16157dfb5..109772661 100644 --- a/nextjs-frontend/components/settings/model.ts +++ b/nextjs-frontend/components/settings/model.ts @@ -815,6 +815,16 @@ export const FIELDS: FieldSpec[] = [ ], hint: "A local model adds full stops, commas and question marks, keeping the ones there; about 2 GB, fetched the first time it’s needed", }, + { + section: "analysis", + key: "cleanup_grammar_url", + label: "LanguageTool server", + kind: "text", + mono: true, + nullable: true, + placeholder: "http://languagetool:8010", + hint: "Grammar and punctuation fixes from LanguageTool, such as the compose files’ languagetool profile: off while empty. Spelling and style are left alone, and so are names. Transcripts are sent to it", + }, { section: "analysis", key: "gazetteer", diff --git a/threat-model.json b/threat-model.json index 9a7ef45d8..d148e1f3e 100644 --- a/threat-model.json +++ b/threat-model.json @@ -73,7 +73,7 @@ "News feed sources", "Connect with Google", "POST /imports/web, extension http tools, notification targets", - "admin settings (model, embeddings, speech, decisions, Fedora, bridge, telemetry, IIIF import, rclone remotes, IMAP, Gmail sign-in, iCal, news feeds, MQTT bridge, OIDC issuer, web search)", + "admin settings (model, embeddings, speech, decisions, Fedora, bridge, telemetry, IIIF import, rclone remotes, IMAP, Gmail sign-in, iCal, news feeds, MQTT bridge, OIDC issuer, web search, LanguageTool server)", "replies from remote servers", ".env, archive.yaml, LENS_*, ARCHIVE_SECRET_KEY, RCLONE_BINARY" ], @@ -493,7 +493,7 @@ "source": "docs/assistant.md \"Mail, calendars and files\"" }, { - "entry_point": "admin settings (model, embeddings, speech, decisions, Fedora, bridge, telemetry, IIIF import, rclone remotes, IMAP, Gmail sign-in, iCal, news feeds, MQTT bridge, OIDC issuer, web search)", + "entry_point": "admin settings (model, embeddings, speech, decisions, Fedora, bridge, telemetry, IIIF import, rclone remotes, IMAP, Gmail sign-in, iCal, news feeds, MQTT bridge, OIDC issuer, web search, LanguageTool server)", "parameter": "URL, credentials, api_key_env", "attacker_controllable": "no", "component": "outbound", @@ -1536,7 +1536,7 @@ }, { "property": "N-EGRESS", - "reason": "an egress firewall; netguard guards web import, iCal and news feeds, extension http tools, notifications, the assistant's web search and page reads, Chromium and the opt-in browser container (as browser-proxy), nothing else; admin-set URLs other than iCal and news feeds (model, speech, Fedora, IIIF import, rclone sources, IMAP, MQTT bridge, OIDC, telemetry, tunnel, a SearXNG instance for web search) are unguarded", + "reason": "an egress firewall; netguard guards web import, iCal and news feeds, extension http tools, notifications, the assistant's web search and page reads, Chromium and the opt-in browser container (as browser-proxy), nothing else; admin-set URLs other than iCal and news feeds (model, speech, Fedora, IIIF import, rclone sources, IMAP, MQTT bridge, OIDC, telemetry, tunnel, a SearXNG instance for web search, LanguageTool server) are unguarded", "false_friend": true, "provenance": "documented", "source": "maintainer ruling, 2026-10" diff --git a/threat-model.md b/threat-model.md index 8996b270d..8401491f1 100644 --- a/threat-model.md +++ b/threat-model.md @@ -302,7 +302,7 @@ classified by gate only *(inferred, Q8)*; unread modules are listed in Q34. | `POST /imports/web`, extension `http` tools, notification targets | URL | yes: editors (web, tools), owners (targets) | resource-name, x-network-destination | none | *(maintainer, 2026-10)* | | the assistant's `web_search` and `web_read` (off until an admin picks a provider, `web.search`) | a search query and a page address, both chosen by the model; the results and pages that come back | yes: anyone who may use the assistant, and indirectly whatever it read, through the model; results and pages: whoever runs the site | resource-name, x-network-destination, x-prompt-injection, size | none; queries go only to the admin's provider; pages are fetched through netguard like web import, 10 MB at most (P-SSRF-GUARD); once the answer has read outside text, a query or address neither the person nor a tool gave it asks first (P-INJECTION-EGRESS); results and pages reach the model marked as data; the ledger keeps the provider or host, never the query or page | *(documented, docs/assistant.md "a search or a page read sends words out of Lens")* | | the assistant's connection tools (`connection_tools.py`: `connections`, `search_mail`, `read_mail`, `calendar_events`, `list_files`, `find_files`, `read_file`, `import_from_connection`, `send_mail`, `add_event`, `write_file`) | a connection's id or name, a Gmail search, a message id, a time span, a path; the mail, events and files that come back; the message, event or text file the model would send or write | yes: admins who use the assistant, and indirectly whatever it read, through the model; mail and events: anyone who can email the account or invite it; files: whoever writes to the storage | resource-name, x-prompt-injection, size | none — admins only, and only the tools for the connections there are; reading changes nothing in the account; a file is read up to 20 MB and a message's or file's text up to 11,000 characters; mail, events and files reach the model marked as data, and once read, tools that could carry it out of Lens ask first (P-INJECTION-EGRESS); bringing anything into a namespace waits for the person's yes; sending mail, saving a draft, adding an event and writing a file always wait for the person's yes, even in a setup conversation, with all of it shown (and a note when the answer read outside text), and the model is told never to act on what it read; sending and adding work only on a Gmail or Google Calendar connection an admin let do so (write, which asks Google for gmail.compose or calendar.events), checked again when approved; at most 50 addresses a field, 50,000 characters a message and 200,000 a file; a file is replaced only when asked and approved, and Google Drive connected read only is never written; each change is in the audit log; the ledger keeps what was called, never what came back | *(documented, docs/assistant.md "Mail, calendars and files")* | -| admin settings: model, embeddings, speech, decisions, Fedora, bridge, Telegram bot, Slack app tokens, assistant mailbox, telemetry, IIIF import, rclone remotes, IMAP, Gmail (a Google refresh token or a Workspace service account key), iCal, news feeds, MQTT bridge, OIDC issuer, web search (a SearXNG address, a Brave Search or Tavily key) | URL, credentials, `api_key_env` | admin only; documented guards still apply (P-ADMIN-GUARD) | resource-name, x-credential | admin is trusted | *(maintainer, 2026-10)* | +| admin settings: model, embeddings, speech, decisions, Fedora, bridge, Telegram bot, Slack app tokens, assistant mailbox, telemetry, IIIF import, rclone remotes, IMAP, Gmail (a Google refresh token or a Workspace service account key), iCal, news feeds, MQTT bridge, OIDC issuer, web search (a SearXNG address, a Brave Search or Tavily key), a LanguageTool server for the clean step's grammar fixes | URL, credentials, `api_key_env` | admin only; documented guards still apply (P-ADMIN-GUARD) | resource-name, x-credential | admin is trusted | *(maintainer, 2026-10)* | | Secret references and the password manager's sign-in: `secrets.*`, a secret field set to `vaultwarden://…` (`PUT /settings/
`, MCP `set_secret_reference`, the assistant's `change_settings`), `GET /secrets`, `GET /secrets/items`, `POST /secrets/test`, MCP `list_secrets` and `list_vault_items`, the assistant's `list_secrets` and `vault_items` | a reference (provider, item, field); the vault's address, account, API key and master password; the items and values the vault sends back | admins only; the vault's answers: whoever can write to Lens's vault account or share items into its folder | resource-name, x-credential, x-network-destination | admin is trusted with the address; a reference is checked against the vault before it's saved, and the vault's own sign-in can't be one; only one folder is read; values go only where the field's own value would, never to the API, tools, approval cards or logs (P-SECRET-REFERENCES); the assistant's changes to the section always ask first | *(documented, docs/secrets.md "People and agents see the reference and whether it could be read, but never the value")* | | replies from remote servers | web page HTML and scripts, IIIF manifests, model replies, OIDC discovery, calendars and their redirects | yes, the remote | data, topology | treat as untrusted | *(inferred, Q17)* | | chat, `/graph/ask`, MCP tools | prompt text; model-chosen tool calls; a conversation's or question's picks (`tools`, `max_tool_calls`, `scope`) and the message ids a version names (`regenerate`, `edit`, `parent`, `PUT /chats/{cid}/head`) | yes; also indirectly through content | data, x-prompt-injection | none; what people say to Lens, in the web app or a chat app, reaches automations as outside text (`chat.message`, `lens.untrusted`); picks only narrow (a tool the model wasn't offered isn't run, an admin's switches still win, a scope naming a namespace the person can't read is a 404, and a limit is a whole number up to `ai.max_tool_calls`), and a message id outside the conversation is a 404 | *(documented, docs/assistant.md "limited to what the person can read")* | @@ -609,7 +609,7 @@ Exported by phase 3.6 (backtest) and de-identified. Closing rows carry their sta | ID | The project does not provide | Conditions / boundary | Tier | False friend? | Provenance | | --- | --- | --- | --- | --- | --- | | N-ADMIN | Protection from the instance admin or host operator, including admin-set URLs reaching internal hosts, admin code, templates, regexes, component installs and `api_key_env` | Covers only operands an admin or operator alone can set. Excludes the guards Lens documents on them (P-ADMIN-GUARD): an escape of one is `VALID`. Also excludes what happens to saved telemetry headers when the endpoint moves to another host; that row is unresolved (Q35), so such a report routes `MODEL-GAP`, not a close. A non-admin reaching any admin operand is in model. | security-critical | no | *(maintainer, 2026-10)* | -| N-EGRESS | An egress firewall. `netguard` guards web import, iCal and news feeds, extension `http` tools, notifications, the assistant's web search and page reads, Chromium and the opt-in browser container (as `browser-proxy`); nothing else | Admin-set URLs other than iCal and news feeds (model, speech, Fedora, IIIF import, rclone sources, IMAP, MQTT bridge, OIDC, telemetry, tunnel, a SearXNG instance for web search) are unguarded | security-critical | yes: "public addresses only" reads as server-wide | *(maintainer, 2026-10)* | +| N-EGRESS | An egress firewall. `netguard` guards web import, iCal and news feeds, extension `http` tools, notifications, the assistant's web search and page reads, Chromium and the opt-in browser container (as `browser-proxy`); nothing else | Admin-set URLs other than iCal and news feeds (model, speech, Fedora, IIIF import, rclone sources, IMAP, MQTT bridge, OIDC, telemetry, tunnel, a SearXNG instance for web search, LanguageTool server) are unguarded | security-critical | yes: "public addresses only" reads as server-wide | *(maintainer, 2026-10)* | | N-CODE-GUARD | A sandbox for Python code tools. Audit hooks, rlimits and an optional network namespace "raise the bar" only | Admin-written code; runs as the Lens user | security-critical | yes: guard is not isolation | *(documented, domain/code_tools.py docstring "they aren't a wall")* | | N-DOWNLOADS | Integrity beyond HTTPS and the publisher for llama.cpp, cloudflared, pip installs and `hf:` models | Catalog GGUF files are still checked (P-MODEL-CHECKSUM) | security-critical | no | *(maintainer, 2026-10)* | | N-DB-PLAIN | Encryption of the SurrealDB content (assets' text, indexes), derived files, renditions, reports, exports and the storage-source cache | Stated limit | security-critical | yes: vaults cover files only | *(documented, docs/encryption.md "the SurrealDB database itself")* | diff --git a/threat-model.yaml b/threat-model.yaml index b053c0410..c77ddcf1a 100644 --- a/threat-model.yaml +++ b/threat-model.yaml @@ -409,7 +409,7 @@ entry_points: control_kinds: [resource-name, x-prompt-injection, size] caller_must_enforce: "none — admins only, and only the tools for the connections there are; reading changes nothing in the account; a file is read up to 20 MB and a message's or file's text up to 11,000 characters; mail, events and files reach the model marked as data, and once read, tools that could carry it out of Lens ask first (P-INJECTION-EGRESS); bringing anything into a namespace waits for the person's yes; sending mail, saving a draft, adding an event and writing a file always wait for the person's yes, even in a setup conversation, with all of it shown (and a note when the answer read outside text), and the model is told never to act on what it read; sending and adding work only on a Gmail or Google Calendar connection an admin let do so (write, which asks Google for gmail.compose or calendar.events), checked again when approved; at most 50 addresses a field, 50,000 characters a message and 200,000 a file; a file is replaced only when asked and approved, and Google Drive connected read only is never written; each change is in the audit log; the ledger keeps what was called, never what came back" provenance: {kind: documented, source: 'docs/assistant.md "Mail, calendars and files"'} - - id: "admin settings (model, embeddings, speech, decisions, Fedora, bridge, telemetry, IIIF import, rclone remotes, IMAP, Gmail sign-in, iCal, news feeds, MQTT bridge, OIDC issuer, web search)" + - id: "admin settings (model, embeddings, speech, decisions, Fedora, bridge, telemetry, IIIF import, rclone remotes, IMAP, Gmail sign-in, iCal, news feeds, MQTT bridge, OIDC issuer, web search, LanguageTool server)" component: outbound parameters: - name: "URL, credentials, api_key_env" @@ -1996,7 +1996,7 @@ properties_claimed: # ---------------------------------------------------------------- §1.12 properties_disclaimed: - {id: N-ADMIN, components: [outbound, ingest, assistant, optional], tier: security-critical, false_friend: false, conditions: "protection from the instance admin or host operator (admin-set URLs reaching internal hosts, admin code, templates, regexes, component installs, api_key_env). Covers only operands an admin or operator alone can set. Excludes the guards Lens documents on them (P-ADMIN-GUARD): an escape of one is VALID. Excludes saved telemetry headers when the endpoint moves host (unresolved, Q35 -> MODEL-GAP). A non-admin reaching any admin operand is in model.", provenance: {kind: maintainer, date: "2026-10"}} - - {id: N-EGRESS, components: [outbound], tier: security-critical, false_friend: true, conditions: "an egress firewall; netguard guards web import, iCal and news feeds, extension http tools, notifications, the assistant's web search and page reads, Chromium and the opt-in browser container (as browser-proxy), nothing else; admin-set URLs other than iCal and news feeds (model, speech, Fedora, IIIF import, rclone sources, IMAP, MQTT bridge, OIDC, telemetry, tunnel, a SearXNG instance for web search) are unguarded", x-false-friend-note: "'public addresses only' reads as server-wide", provenance: {kind: maintainer, date: "2026-10"}} + - {id: N-EGRESS, components: [outbound], tier: security-critical, false_friend: true, conditions: "an egress firewall; netguard guards web import, iCal and news feeds, extension http tools, notifications, the assistant's web search and page reads, Chromium and the opt-in browser container (as browser-proxy), nothing else; admin-set URLs other than iCal and news feeds (model, speech, Fedora, IIIF import, rclone sources, IMAP, MQTT bridge, OIDC, telemetry, tunnel, a SearXNG instance for web search, LanguageTool server) are unguarded", x-false-friend-note: "'public addresses only' reads as server-wide", provenance: {kind: maintainer, date: "2026-10"}} - {id: N-CODE-GUARD, components: [assistant], tier: security-critical, false_friend: true, conditions: "a sandbox for Python code tools; audit hooks, rlimits and an optional network namespace only raise the bar; admin-written code; runs as the Lens user", x-false-friend-note: "guard is not isolation", provenance: {kind: documented, source: 'domain/code_tools.py docstring "they aren''t a wall"'}} - {id: N-DOWNLOADS, components: [optional], tier: security-critical, false_friend: false, conditions: "integrity beyond HTTPS and the publisher for llama.cpp, cloudflared, pip installs and hf: models; catalog GGUF files are still checked (P-MODEL-CHECKSUM)", provenance: {kind: maintainer, date: "2026-10"}} - {id: N-DB-PLAIN, components: [outbound], tier: security-critical, false_friend: true, conditions: "encryption of the SurrealDB content (assets' text, indexes), derived files, renditions, reports, exports and the storage-source cache; stated limit", x-false-friend-note: "vaults cover files only", provenance: {kind: documented, source: 'docs/encryption.md "the SurrealDB database itself"'}}