Skip to content

feat(analysis): fix transcripts' grammar with a LanguageTool server - #521

Merged
adeelahmad merged 1 commit into
mainfrom
claude/project-thread-hyaa6h
Oct 18, 2026
Merged

adeelahmad merged 1 commit into
mainfrom
claude/project-thread-hyaa6h

Conversation

@adeelahmad

Copy link
Copy Markdown
Owner

Requested by Adeel · project thread

Description

Before: the clean step fixed stutters and spacing but not grammar; the pasted script's LanguageTool pass ("he go" → "he goes") had no place in Lens.

After: with transcripts cleaned up and Settings → Analysis → LanguageTool server set (none by default), the clean step sends a transcript's lines to that server and makes its grammar, punctuation and typography fixes and "a"/"an" ("he go" → "he goes", "a apple" → "an apple"). Spelling, style, wording and repeated-word suggestions are left out, and so is any fix to a word with a capital, so names and how people talk stay as said. Each change is in Change history with Revert. When the server doesn't answer, the rules' changes stand and the job's log says so. The compose files gain an opt-in languagetool profile (docker compose --profile languagetool up -d) that starts one at http://languagetool:8010.

How: cleanup.grammar batches lines (10,000 characters a call) to POST /v2/check, maps LanguageTool's UTF-16 offsets back to each line, follows no redirect, caps the answer's size, and logs each call in the activity ledger. language_tool_python, which the script used, is GPL-3.0 and starts Java itself, so Lens talks to a server instead. Categories were checked against a real LanguageTool 6.8.

Motivation and Context

Last of the series from Adeel's pasted punctuation and grammar script (#476 cleanup, #485 punctuation, #492 spaCy, #504 dates).

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change; ! in the title)

Checklist:

  • The title is a Conventional Commit.
  • My code follows the code style of this project (make hooks checks it on commit).
  • I added or updated tests.
  • My change requires documentation updates, and I have updated docs/ accordingly.
  • Users need more than the title to understand it, so I wrote it under ## Unreleased in CHANGELOG.md.

Done questions (AGENTS.md)

  • Goal: grammar correction in the cleaning step, as the script did; merged on green CI.
  • Agents: PUT /api/v1/settings/analysis (cleanup_grammar_url) and change_settings; changes in the edit history API.
  • Zero setup: optional; nothing changes until a server is set, and the compose profile starts one with one command.
  • Fewest clicks: one field with the address as placeholder.
  • Configurable: Settings → Analysis, default none; the address is validated (http(s), no password).
  • Light: no Java in Lens; the server is an opt-in compose profile (about 1 GB of memory, documented).
  • Private and secure: the default profile runs on Lens's own network; the lines go only to the set address, with no redirects (threat model's settings row, entry points and N-EGRESS updated).
  • Cost: no LLM calls; each LanguageTool call is in the activity ledger.
  • User decides: off by default; only grammar categories are taken; people's corrections outrank it; every change has Revert.
  • Upgrades: no stored-data change.
  • Honest: docs say which fixes are taken and which aren't; README built list and a docs/requirements.md row added (make requirements passes).
  • Tests in tests/domain/test_cleanup.py (a fake server counting UTF-16 offsets, the categories taken and left, the server down, the address check); docs/processing.md, docs/requirements.md, README, CHANGELOG, DEVLOG updated; OpenAPI unchanged.
  • Open: LanguageTool's HTTP API; the server image is LGPL and runs outside Lens, opt-in, so Lens stays MIT.
  • Reported in the project thread; refine later: start the profile from Settings through the machine agent, per-namespace grammar, more categories as opt-in switches.
  • Reuse: LanguageTool itself, through its HTTP API and its published Docker image; the clean step's edit history and the activity ledger.

Local CI (ci/local-ci.sh): backend (embedded and server SurrealDB), Next.js, pre-commit, title and the upgrade check passed.


Generated by Claude Code

@adeelahmad adeelahmad self-assigned this Oct 17, 2026
@adeelahmad
adeelahmad force-pushed the claude/project-thread-hyaa6h branch 8 times, most recently from 576dfad to 3b83220 Compare October 18, 2026 07:59
@adeelahmad
adeelahmad force-pushed the claude/project-thread-hyaa6h branch from 3b83220 to 7587f43 Compare October 18, 2026 08:35

Copy link
Copy Markdown
Owner Author

Clicks and keystrokes per feature failed on 7587f43: the phone journey reported "worse: chat-ask fails now" (on the previous head it was first-question, the same journey pair). This PR changes the clean step, settings, docs and the compose files; it touches no chat code, and the phone's computer-sized twin passed in the same run. Main's own Clicks run on the base commit cf961f3 passed. #582 traces this flake to a race in a new conversation's first read of GET /chats/{id} (reproduced there in 1 of 7 runs) and fixes it. Re-running the failed job once; if it fails again, I'll carry #582's change into this PR.


Generated by Claude Code

With transcripts cleaned up and Settings > Analysis > LanguageTool server
set (none by default), the clean step sends a transcript's lines to that
server, a batch at a time, and makes its grammar and punctuation fixes ("he
go" > "he goes", "a apple" > "an apple"). Spelling, style and wording
suggestions are left out, and so is any fix to a word with a capital, so
names stay as said. Offsets are read in UTF-16 units, as LanguageTool counts
them. No redirect is followed, each call is an activity ledger row, and when
the server doesn't answer the rules' changes stand and the job log says so.

The compose files gain an opt-in `languagetool` profile
(erikvl87/languagetool 6.8, on Lens's network only). language_tool_python,
which the pasted script used, is GPL-3.0 and starts Java itself, so Lens
talks to a server instead.

Docs, changelog, dev log and threat model updated.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014z3r72SB2quWGKbgGt46mZ
@adeelahmad
adeelahmad force-pushed the claude/project-thread-hyaa6h branch from 7587f43 to 8a83fe7 Compare October 18, 2026 11:06
@adeelahmad
adeelahmad merged commit c6f82c6 into main Oct 18, 2026
22 checks passed
@adeelahmad
adeelahmad deleted the claude/project-thread-hyaa6h branch October 18, 2026 11:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants