Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions DEVLOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,12 @@ from a LanguageTool server, spaCy fetched for named entities, and dates found an

Refine later: revert a whole cleanup at once; cleanup rules for other languages' fillers; per-namespace switch.

Grammar: the clean step takes a LanguageTool server's GRAMMAR, PUNCTUATION and TYPOGRAPHY fixes, and EN_A_VS_AN (filed
under MISC) (analysis.cleanup_grammar_url, the compose files' `languagetool` profile), never spelling, style or a word
with a capital. language_tool_python from the script is GPL-3.0 and starts Java itself, so Lens talks to a server
instead. Refine later: start the profile from Settings through the machine agent; per-namespace grammar; LanguageTool's
other categories as opt-in switches.

## 2026-10-17 · Describing pictures with Moondream on this machine

Moondream is the third describe engine (`llm.describe_engine` `moondream`), opt-in, through Hugging Face transformers
Expand Down
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -131,8 +131,8 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p
asset, a topic or an entity, in a Similar tab on every asset or by selecting text anywhere, and, with anytopdf, photos
by what they show. Pages and shots are described in words by the model provider's vision model or, off by default, by
Florence-2 or Moondream on your own server. Transcripts can be cleaned up before analysis (stutters, "um"s, spacing
and, with a local model, punctuation put back), entities found by spaCy instead of the rules, and the dates people say
read as days on the calendar, each off by default.
and, with a local model, punctuation put back, and grammar by a LanguageTool server you run), entities found by spaCy
instead of the rules, and the dates people say read as days on the calendar, each off by default.
([Graph](docs/graph.md), [entity pages](docs/notes.md#entity-pages),
[similar content](docs/processing.md#similar-content), [graph history](docs/graph-history.md),
[topics](docs/topics.md), [video](docs/video.md), [NSFW classification](docs/nsfw.md),
Expand Down
5 changes: 5 additions & 0 deletions changelog.d/521.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
- **Grammar fixed by LanguageTool, when you run one.** With transcripts cleaned up and Settings → Analysis →
LanguageTool server set (none by default; the compose files' new `languagetool` profile starts one at
`http://languagetool:8010`), the clean step makes its grammar and punctuation fixes too, leaving spelling, style and
names alone. Each change is in Change history with Revert; without the server the rules run alone
([Cleaning up transcripts](docs/processing.md#cleaning-up-transcripts)).
13 changes: 13 additions & 0 deletions docker-compose.prod.yml
Original file line number Diff line number Diff line change
Expand Up @@ -202,6 +202,19 @@ services:
restart: unless-stopped
networks: [lens]

# LanguageTool (docs/processing.md#cleaning-up-transcripts): grammar and punctuation fixes for the clean step. Opt-in,
# about 1 GB of memory: `docker compose --profile languagetool up -d` (or COMPOSE_PROFILES=languagetool in .env), then
# Settings → Analysis → LanguageTool server: http://languagetool:8010. Reachable only inside Lens's network.
languagetool:
image: erikvl87/languagetool:${LANGUAGETOOL_VERSION:-6.8}
profiles: [languagetool]
security_opt: ["no-new-privileges:true"]
environment:
Java_Xms: 256m
Java_Xmx: ${LANGUAGETOOL_HEAP:-768m}
restart: unless-stopped
networks: [lens]

# The MCP runner (mcp-runner/runner.py, docs/assistant.md): MCP servers that run as a program (stdio, usually npx or
# uvx), which admins add as extensions. Opt-in: `docker compose --profile mcp up -d`, or COMPOSE_PROFILES=mcp in .env.
# The sandbox: none of Lens's data or keys, no capabilities, a read-only image, and a network shared only with the
Expand Down
12 changes: 12 additions & 0 deletions docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -173,6 +173,18 @@ services:
JAVA_TOOL_OPTIONS: -Xmx${TIKA_HEAP:-512m}
networks: [lens]

# LanguageTool (docs/processing.md#cleaning-up-transcripts): grammar and punctuation fixes for the clean step. Opt-in,
# about 1 GB of memory: `docker compose --profile languagetool up -d` (or COMPOSE_PROFILES=languagetool in .env), then
# Settings → Analysis → LanguageTool server: http://languagetool:8010. Reachable only inside Lens's network.
languagetool:
image: erikvl87/languagetool:${LANGUAGETOOL_VERSION:-6.8}
profiles: [languagetool]
security_opt: ["no-new-privileges:true"]
environment:
Java_Xms: 256m
Java_Xmx: ${LANGUAGETOOL_HEAP:-768m}
networks: [lens]

# The MCP runner (mcp-runner/runner.py, docs/assistant.md): MCP servers that run as a program (stdio, usually npx or
# uvx), which admins add as extensions. Opt-in: `docker compose --profile mcp up -d`, or COMPOSE_PROFILES=mcp in .env.
# The sandbox: none of Lens's data or keys, no capabilities, a read-only image, and a network shared only with the
Expand Down
12 changes: 10 additions & 2 deletions docs/processing.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,8 +27,8 @@ Every step takes `--ns`, `--limit` and `--force`; `run` does them all, and a loc

Speech comes out of transcription as it was said, and the extractor, chapters, keywords and search all read it. With
`analysis.cleanup` on (Settings → Analysis → Clean up transcripts before analysis; off by default), the `clean` step,
which the standard pipeline runs just before `analyze`, tidies each line of a transcript with rules. It needs no model,
fetches nothing and sends nothing out:
which the standard pipeline runs just before `analyze`, tidies each line of a transcript with rules. The rules need no
model, fetch nothing and send nothing out:

- a word said three times or more in a row is kept once ("I I I looked" → "I looked"), and a doubled pronoun, article
or conjunction ("the the", "I, I") once; "had had", "that that" and "very very" stay;
Expand All @@ -44,6 +44,14 @@ fetches nothing and sends nothing out:
each line marks once. The model, Transformers and PyTorch (about 2 GB) are fetched the first time a clean step
needs them ([Components](components.md)); until then, and in other languages, the rules run alone and the job's
log says why.
- grammar (`analysis.cleanup_grammar_url`, LanguageTool server; none): a [LanguageTool](https://languagetool.org)
server's grammar and punctuation fixes are made too ("he go" → "he goes", "a apple" → "an apple"), and only those:
spelling, style, wording and repeated-word ("very very") suggestions are left out, and so is any fix to a word
with a capital in it, so names stay as they were said. Start one with the compose files' `languagetool` profile
(`docker compose --profile languagetool up -d`, about 1 GB of memory) and set `http://languagetool:8010`. The
transcript's lines go to that address and nowhere else (no redirect is followed), so a server outside your network
sees them; each call is in the activity ledger. When it doesn't answer, the rules' changes stand and the job's log
says so.

These are English rules ("um" is a word in German and Portuguese), so they apply to English lines: the line's or the
asset's language when transcription gave one, else when the text reads as English. In other languages only a word
Expand Down
1 change: 1 addition & 0 deletions docs/requirements.md
Original file line number Diff line number Diff line change
Expand Up @@ -160,6 +160,7 @@ request, to another check that proves the same thing; loosening a requirement or
| Or by Moondream on this server, off by default, fetched at a pinned revision only where the chosen model can run, and saying why where it can't | [test_moondream.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_moondream.py) `test_moondream_describes_pictures_without_a_model_provider`, `test_where_each_moondream_can_run`, `test_on_a_cpu_moondream_3_is_never_fetched_and_the_step_says_why`, `test_moondream_is_fetched_pinned`, `test_admins_and_agents_choose_moondream` |
| Transcripts cleaned up before analysis (stutters, "um"s, spacing), off by default, each change in the edit history with Revert and people's corrections left alone | [test_cleanup.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_cleanup.py) `test_off_by_default_changes_nothing`, `test_clean_step_tidies_lines_and_keeps_them_in_the_history`, `test_people_s_corrections_outrank_the_rules` |
| Punctuation put back in transcripts that lack it by a local model, fetched only when picked | [test_cleanup.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_cleanup.py) `test_punctuation_put_back_where_it_is_missing`, `test_without_the_model_the_rules_run_alone`, `test_the_model_is_fetched_only_when_it_is_asked_for` |
| Grammar fixed by a LanguageTool server you run, when its address is set: grammar fixes only, never spelling or a word with a capital | [test_cleanup.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_cleanup.py) `test_grammar_fixes_from_languagetool_leave_names_and_spelling`, `test_without_its_server_the_rules_run_alone`, `test_the_grammar_server_is_an_address_without_a_password` |
| Entities found by spaCy, fetched the first time it's picked, with the defined entities first and the rules until it's there | [test_spacy_entities.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_spacy_entities.py) `test_defined_entities_come_first_then_spacy_s_names`, `test_the_analyze_step_and_workflows_use_it`, `test_without_spacy_the_rules_find_them`, `test_spacy_is_fetched_only_when_it_is_picked` |
| Dates said in an asset read as days ("next Tuesday", "last week"), off by default, with dates that read two ways left out | [test_dates.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_dates.py) `test_dates_are_read_as_days`, `test_what_could_be_two_dates_is_left_out`, `test_off_by_default_reads_none` |

Expand Down
136 changes: 128 additions & 8 deletions fastapi_backend/app/domain/cleanup.py
Original file line number Diff line number Diff line change
Expand Up @@ -22,16 +22,26 @@
once. The model and PyTorch (about 2 GB) are fetched the first time a clean step needs them (components.py); until
then, or where it can't run, the rules run alone.

Grammar (`analysis.cleanup_grammar_url`, none by default): with a LanguageTool server's address (the compose files'
`languagetool` profile: http://languagetool:8010), its grammar and punctuation fixes are made too ("he go" → "he goes",
"a apple" → "an apple"), and only those: spelling, style, wording and repeated-word suggestions are left out, and so is
any fix to a word with a capital in it, so names stay as they were said. A line goes to that server and nowhere else; when it
doesn't answer, the rules' changes stand alone.

Every line it changes is kept in the asset's edit history, by "text cleanup": Change history lists them and Revert
puts a line back. A line someone corrected is never cleaned, since people's corrections outrank the rules, and running
the rules again on what they made changes nothing.
"""

from __future__ import annotations

import json
import re
import urllib.error
import urllib.parse
import urllib.request

from . import store
from . import activity, store

R = store.R
CLEANER = "text cleanup" # who the edit history says changed a line
Expand Down Expand Up @@ -258,6 +268,104 @@ def _punctuation(cfg, rec, segs):
return texts, [segs[k]["id"] for k in todo], f"punctuation restored by {PUNCTUATION_MODEL[0]}"


# ---------- grammar, by a LanguageTool server ----------
# LanguageTool's fixes taken: its grammar, punctuation and typography categories, and "a" or "an", which it files under
# MISC. Spelling (TYPOS), style, wording and its repeated-word rule ("very very") would rewrite how people talk.
GRAMMAR = {"GRAMMAR", "PUNCTUATION", "TYPOGRAPHY"}
GRAMMAR_RULES = {"EN_A_VS_AN"}
GRAMMAR_CHARS = 10_000 # text sent to the server at once
GRAMMAR_ANSWER = 4 << 20 # the most of an answer read
GRAMMAR_SECONDS = 60


def _check(url, text, lang, cfg):
"""LanguageTool's matches for `text` ([{offset, length, replacements, rule}]); Unavailable when it doesn't answer."""
body = urllib.parse.urlencode({"text": text, "language": lang or "auto"}).encode()
req = urllib.request.Request(f"{url}/v2/check", data=body, headers={"Accept": "application/json"}, method="POST")
host = urllib.parse.urlsplit(url).hostname
try:
with activity.call("grammar.check", cfg, detail={"host": host, "chars": len(text)}):
with _opener.open(req, timeout=GRAMMAR_SECONDS) as r:
got = json.loads(r.read(GRAMMAR_ANSWER))
except (OSError, ValueError) as e:
raise Unavailable(f"the LanguageTool server at {host} didn't answer ({e})") from None
return got.get("matches") or [] if isinstance(got, dict) else []


class _NoRedirect(urllib.request.HTTPRedirectHandler):
def redirect_request(self, *args, **kwargs): # the text goes to the address set, and nowhere it sends it on
return None


_opener = urllib.request.build_opener(_NoRedirect)


def _u16(text):
return len(text.encode("utf-16-le")) // 2


def _cp(text, units):
"""A UTF-16 offset into `text` (LanguageTool's, from Java) as Python's."""
if text.isascii():
return units
n = 0
for i, ch in enumerate(text):
if n >= units:
return i
n += 2 if ord(ch) > 0xFFFF else 1
return len(text)


def _fix(text, matches):
"""`text` with the fixes taken (GRAMMAR, GRAMMAR_RULES): the first replacement of each, none to a word with a
capital, and none that overlaps another."""
upto = len(text)
for m in sorted(matches, key=lambda m: -m["offset"]):
a, b = _cp(text, m["offset"]), _cp(text, m["offset"] + m["length"])
rule = m.get("rule") or {}
cat = ((rule.get("category") or {}).get("id") or "").upper()
reps = [r.get("value") for r in m.get("replacements") or [] if isinstance(r.get("value"), str)]
taken = cat in GRAMMAR or rule.get("id") in GRAMMAR_RULES
if not taken or not reps or a < 0 or b > upto or any(c.isupper() for c in text[a:b] + reps[0]):
continue
text, upto = text[:a] + reps[0] + text[b:], a
return text


def grammar(cfg, texts, lang=None):
"""The texts with a LanguageTool server's grammar and punctuation fixes (analysis.cleanup_grammar_url), sent a
batch of lines at a time. Unavailable when no server is set or it doesn't answer."""
url = ((cfg.get("analysis") or {}).get("cleanup_grammar_url") or "").rstrip("/")
if not url:
raise Unavailable("no LanguageTool server is set")
batches, batch, size = [], [], 0
for k, t in enumerate(texts):
if not t.strip():
continue
if batch and size + len(t) + 2 > GRAMMAR_CHARS:
batches.append(batch)
batch, size = [], 0
batch.append(k)
size += len(t) + 2
batches += [batch] if batch else []
lang = lang if lang and lang not in ("auto", "nospeech") else "auto"
out = list(texts)
for batch in batches: # a batch's lines one after another, a blank line between them
joined, starts, at = "", [], 0
for j in batch:
starts.append(at) # where each line starts, in UTF-16 units as LanguageTool counts
joined += texts[j] + "\n\n"
at += _u16(texts[j]) + 2
found = _check(url, joined, lang, cfg)
for j, at in zip(batch, starts, strict=True):
end = at + _u16(texts[j])
mine = [
{**m, "offset": m["offset"] - at} for m in found if at <= m.get("offset", -1) and m["offset"] + m.get("length", 0) <= end
]
out[j] = _fix(texts[j], mine)
return out


# ---------- an asset's lines ----------
def plan(db, cfg, rid):
"""What cleaning an asset does: {changes: [(segment id, idx, text, cleaned)], punctuated: [segment ids], note}.
Expand All @@ -272,20 +380,32 @@ def plan(db, cfg, rid):
fillers = bool((cfg.get("analysis") or {}).get("cleanup_fillers", True))
whole = english(rec.get("language"), [s.get("text") or "" for s in segs[:400]])
marked, punctuated, note = _punctuation(cfg, rec, segs)
out, prev = [], None
for s, text in zip(segs, marked):
cleaned, prev = [], None
for s, text in zip(segs, marked, strict=True):
ended = prev is None or prev["speaker"] != s.get("speaker") or SENTENCE_END.search(prev["text"]) is not None
prev = {"speaker": s.get("speaker"), "text": text}
if s["idx"] in people or not text.strip():
cleaned.append(None)
continue
lang = s.get("lang")
en = english(lang, [text]) if lang and lang not in ("auto", "nospeech") else whole
new = clean_line(text, en, fillers, ended)
if new and new != (s.get("text") or ""):
out.append((s["id"], s["idx"], s.get("text") or "", new))
prev["text"] = new
prev["text"] = clean_line(text, en, fillers, ended) or text
cleaned.append(prev["text"])
notes = [note] if note else []
if (cfg.get("analysis") or {}).get("cleanup_grammar_url"):
todo = [k for k, t in enumerate(cleaned) if t]
try:
fixed = grammar(cfg, [cleaned[k] for k in todo], rec.get("language"))
for k, t in zip(todo, fixed, strict=True):
cleaned[k] = t
notes.append("grammar checked by LanguageTool")
except Unavailable as e:
notes.append(f"grammar not checked: {e}")
out = [
(s["id"], s["idx"], s.get("text") or "", new) for s, new in zip(segs, cleaned, strict=True) if new and new != (s.get("text") or "")
]
kept = {s["id"] for s in segs if s["idx"] in people}
return {"changes": out, "punctuated": [i for i in punctuated if i not in kept], "note": note}
return {"changes": out, "punctuated": [i for i in punctuated if i not in kept], "note": "; ".join(notes) or None}


def clean_recording(db, cfg, rid, say=None):
Expand Down
9 changes: 9 additions & 0 deletions fastapi_backend/app/domain/settings.py
Original file line number Diff line number Diff line change
Expand Up @@ -632,6 +632,15 @@ def _check(section, key, value, default):
if not (isinstance(value, str) and VIEWER_URL.match(value.strip())):
raise ValueError("documents.anytopdf_url is a conversion node's http(s) address, such as https://convert.home:8640")
return value.strip().rstrip("/")
if (section, key) == ("analysis", "cleanup_grammar_url"):
if value in (None, ""):
return None
u = urllib.parse.urlsplit(value.strip()) if isinstance(value, str) else None
if not (u and VIEWER_URL.match(value.strip()) and u.hostname and u.username is None and u.password is None):
raise ValueError(
"analysis.cleanup_grammar_url is a LanguageTool server's http(s) address with no password in it, such as http://languagetool:8010"
)
return value.strip().rstrip("/")
if (section, key) == ("documents", "tika_url"):
if value in (None, ""):
return None
Expand Down
6 changes: 4 additions & 2 deletions fastapi_backend/app/domain/store.py
Original file line number Diff line number Diff line change
Expand Up @@ -156,8 +156,9 @@ def labels():
# anytopdf: the places and dates anytopdf (run here) finds, as hints for the extractor (enrich.py); dates: the dates
# each line gives kept as days on the calendar (dates.py); cleanup: the
# clean step tidies transcripts before analysis (cleanup.py), cleanup_fillers taking out "um" and "uh" too, and
# cleanup_punctuation (off, missing, always) putting marks back with a local model; typing: which model types the
# names the rules leave terms (name_types.asking)
# cleanup_punctuation (off, missing, always) putting marks back with a local model, cleanup_grammar_url a
# LanguageTool server's grammar and punctuation fixes (the compose files' `languagetool` profile); typing: which
# model types the names the rules leave terms (name_types.asking)
"analysis": {
"entities": "rules",
"typing": "auto",
Expand All @@ -168,6 +169,7 @@ def labels():
"cleanup": False,
"cleanup_fillers": True,
"cleanup_punctuation": "off",
"cleanup_grammar_url": None,
},
# chat_models: the models people may pick in Chat; empty: whatever the model server lists (docs/configuration.md)
"llm": {
Expand Down
Loading
Loading