Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,10 @@ The backend (`fastapi_backend`) and the frontend (`nextjs-frontend`) are version

## Unreleased

- **Apache Tika.** Lens can use an Apache Tika server (`--profile tika` in the compose files, or one of your own, set
in Settings → Documents). Documents nothing here can make into a PDF, such as Word files on a lean server or a
Raspberry Pi, are read by Tika as text, and every document's own title, authors, language and dates are read as
Dublin Core terms. Off until a Tika server is set ([Configuration](docs/configuration.md#apache-tika)).
- **New pipelines, workflows, templates and API tokens come with a name.** The web app guesses one from what you're
making ("Transcribe to report", "Recording workflow", "Prompt template 2", "Read only token"), as it already did for
routines, so making one takes no typing; type a name to choose your own. The pencil next to the name renames it
Expand Down
2 changes: 1 addition & 1 deletion ci/clicks/run.sh
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ stop() {
wait "$p" 2>/dev/null || true
kill "$w" 2>/dev/null || true
done
rm -rf "$tmp"
rm -rf "$tmp" 2>/dev/null || { sleep 2; rm -rf "$tmp" || true; }
}
trap stop EXIT INT TERM

Expand Down
15 changes: 15 additions & 0 deletions docker-compose.prod.yml
Original file line number Diff line number Diff line change
Expand Up @@ -164,6 +164,21 @@ services:
restart: unless-stopped
networks: [lens]

# Apache Tika (docs/configuration.md#apache-tika): reads the text and description of about a thousand kinds of file where Lens can't make
# them into PDFs itself, and what every document says about itself. Opt-in, about 250 MB of memory:
# `docker compose --profile tika up -d` (or COMPOSE_PROFILES=tika in .env), then Settings → Documents → Tika server:
# http://tika:9998. Reachable only inside Lens's network; it has no sign-in of its own.
tika:
image: apache/tika:${TIKA_VERSION:-4.0.0-1}
profiles: [tika]
security_opt: ["no-new-privileges:true"]
read_only: true
tmpfs: [/tmp]
environment:
JAVA_TOOL_OPTIONS: -Xmx${TIKA_HEAP:-512m}
restart: unless-stopped
networks: [lens]

# The MCP runner (mcp-runner/runner.py, docs/assistant.md): MCP servers that run as a program (stdio, usually npx or
# uvx), which admins add as extensions. Opt-in: `docker compose --profile mcp up -d`, or COMPOSE_PROFILES=mcp in .env.
# The sandbox: none of Lens's data or keys, no capabilities, a read-only image, and a network shared only with the
Expand Down
14 changes: 14 additions & 0 deletions docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -145,6 +145,20 @@ services:
- matterbridge-data:/matterbridge
networks: [lens]

# Apache Tika (docs/configuration.md#apache-tika): reads the text and description of about a thousand kinds of file where Lens can't make
# them into PDFs itself, and what every document says about itself. Opt-in, about 250 MB of memory:
# `docker compose --profile tika up -d` (or COMPOSE_PROFILES=tika in .env), then Settings → Documents → Tika server:
# http://tika:9998. Reachable only inside Lens's network; it has no sign-in of its own.
tika:
image: apache/tika:${TIKA_VERSION:-4.0.0-1}
profiles: [tika]
security_opt: ["no-new-privileges:true"]
read_only: true
tmpfs: [/tmp]
environment:
JAVA_TOOL_OPTIONS: -Xmx${TIKA_HEAP:-512m}
networks: [lens]

# The MCP runner (mcp-runner/runner.py, docs/assistant.md): MCP servers that run as a program (stdio, usually npx or
# uvx), which admins add as extensions. Opt-in: `docker compose --profile mcp up -d`, or COMPOSE_PROFILES=mcp in .env.
# The sandbox: none of Lens's data or keys, no capabilities, a read-only image, and a network shared only with the
Expand Down
5 changes: 4 additions & 1 deletion docs/api.md
Original file line number Diff line number Diff line change
Expand Up @@ -509,7 +509,10 @@ made, unless someone named or dated it. Its attachments are kept as its files (r
read become resources of their own beside it, each with `attached_to: {resource, file, title}` (the email, and its
title when the caller may see it); an attachment file's `resource` is the resource it became, or the one it was already
in the namespace (`documents.attachment_resources`). Transcribing an email again keeps and makes nothing twice.
`GET /uploads/limits` says what the server can convert (`convert: {office, pages, msg}`); an upload it can't read is
`GET /uploads/limits` says what the server can convert (`convert: {office, pages, msg, tika}`); with an
[Apache Tika](configuration.md#apache-tika) server set, what nothing else can convert is read by Tika as text without
pages (`engine` `text:tika`), and every document has `described`, what it says about itself as Dublin Core terms
(`POST /settings/documents/tika/test` checks the server). An upload it can't read is
refused, saying what it needs, and the files that can be transcripts too (PDFs, Word, text and Markdown) can be imported
as transcripts instead (`POST /import`). Its pipeline's read step (its transcribe step, for a document or an image:
no speech model is fetched for one) draws its pages and reads their text: a PDF's own text, block by block with where each block is on its page (poppler's
Expand Down
34 changes: 34 additions & 0 deletions docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -237,6 +237,9 @@ each the first time it's needed; without them, PDFs and images are read as befor
| `documents.converter` | `auto` | who makes the PDFs: `auto` uses [anytopdf](#anytopdf) only for what LibreOffice and Chromium can't do here, `anytopdf` uses it for every document and image (and fetches it), `lens` never uses it |
| `documents.anytopdf_url` | none | a conversion node: another machine running `anytopdf queue serve` and `anytopdf queue work`, such as `https://convert.home:8640`. Set, it converts instead of the program here |
| `documents.anytopdf_token` | none | the node's bearer token (its `ANYTOPDF_QUEUE_TOKEN`), kept encrypted |
| `documents.tika_url` | none (off) | an [Apache Tika](#apache-tika) server, such as `http://tika:9998` for the compose files' `tika` service |
| `documents.tika_text` | true | with a Tika server: documents nothing here can make into a PDF are read by Tika, as text without pages |
| `documents.tika_metadata` | true | with a Tika server: what every document says about itself (title, authors, language, dates) is read by Tika |

Set at startup only (the config file; the web app can't choose what the server runs):

Expand All @@ -247,6 +250,37 @@ Set at startup only (the config file; the web app can't choose what the server r
| `documents.anytopdf` | the one Lens downloaded, else `anytopdf` on PATH | the anytopdf program |
| `documents.web_networks` | `[]` | networks (CIDR, such as `10.20.0.0/16`) that web pages and calendar feeds may be fetched from besides the public internet: for an intranet, or a calendar server at home; loopback and cloud metadata addresses stay out unless listed. `LENS_WEB_NETWORKS` in `.env` (comma-separated) adds to it, for Docker and the packages |

### Apache Tika

[Apache Tika](https://tika.apache.org) (Apache-2.0) reads the text and the description of about a thousand kinds of
file. Lens can use a Tika server; it's off until `documents.tika_url` is set. The compose files have one ready, as an
opt-in profile (about 250 MB of memory, so leave it off on a small Raspberry Pi):

```sh
docker compose -f docker-compose.prod.yml --profile tika up -d
```

(or add `COMPOSE_PROFILES=tika` to Lens's `.env` and run the install command again). Then, in Settings → Documents,
set the Tika server to `http://tika:9998` and press Test. A Tika server elsewhere works the same way.

- **Reading what nothing else can** (`documents.tika_text`). When neither LibreOffice, Chromium nor
[anytopdf](#anytopdf) can make a document into a PDF here (a Word file on a lean server, say), or one of them fails
on it, Tika reads it instead. Uploads and folder scans then take every document type, Office files included. Such
a document has its text, which is searched, analyzed and asked about like any other's, but no pages to look at;
its `engine` is `text:tika`. An email or an archive read this way gives its text, including its attachments' and
files', but they aren't kept as files or resources of their own.
- **What a document says about itself** (`documents.tika_metadata`). Every document is also described by Tika, as
Dublin Core terms (`GET /resources/{id}` → `described`: title, creators, subject, description, publisher,
language, created, modified, pages, words, format). Its title and creation date become the resource's while it's
still named and dated after its file (an email's own subject and date come first), and its authors and language
are its [metadata](rdf.md)'s until someone saves others. If Tika can't describe it, the document is read anyway.

The document goes only to the address set: http(s) with no user name or password in it, no redirect followed, and
sent as `document.<type>`, never by its own name. Tika has no sign-in of its own, so the compose service is reachable
only inside Lens's network; put one elsewhere behind TLS on a network you trust, since every document is sent to it.
Its answers are bounded (64 MB) before Lens reads them. The compose service runs read-only, as a user without
privileges, with `TIKA_HEAP` (default `512m`) as its Java heap and `TIKA_VERSION` (default `4.0.0-1`) as its image.

### anytopdf

[anytopdf](https://github.com/adeelahmad/anytopdf-rs) (MIT or Apache-2.0) is Lens's sister project: one static program
Expand Down
8 changes: 5 additions & 3 deletions docs/deployment.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,12 @@
# Deployment

Lens runs as containers: SurrealDB, the API, one or more workers, and the Next.js app. `docker-compose.prod.yml`
is a production-shaped starting point. Two more services are opt-in profiles: `--profile fedora` adds a Fedora 6
repository to keep a copy of the archive in ([Fedora](fedora.md)), and `--profile opensearch` adds OpenSearch for word
is a production-shaped starting point. More services are opt-in profiles: `--profile fedora` adds a Fedora 6
repository to keep a copy of the archive in ([Fedora](fedora.md)), `--profile opensearch` adds OpenSearch for word
search on a big archive (then choose it in Settings → Search and reindex; the default is a SQLite index in
`data_dir/search`).
`data_dir/search`), and `--profile tika` adds an Apache Tika server that reads documents nothing else here can, and
what every document says about itself (then set it in Settings → Documents;
[Configuration](configuration.md#apache-tika)).

```bash
export ACCESS_SECRET_KEY=$(openssl rand -hex 32) ARCHIVE_SECRET_KEY=$(openssl rand -hex 32) \
Expand Down
13 changes: 13 additions & 0 deletions fastapi_backend/app/api/v1/routes/admin.py
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,7 @@
store,
telegram,
telemetry,
tika,
tunnel,
)
from app.schemas.admin import (
Expand Down Expand Up @@ -73,6 +74,7 @@
TelegramStatus,
TelemetryStatus,
TelemetryTestResult,
TikaTestResult,
TunnelStatus,
)
from app.schemas.auth import AccountToken
Expand Down Expand Up @@ -281,6 +283,17 @@ def test_conversion_node(user: AdminWriter, cfg: Cfg) -> ConversionNodeTestResul
return ConversionNodeTestResult(ok=error is None, error=error, ms=None if error else int((time.time() - t0) * 1000))


@router.post("/settings/documents/tika/test")
def test_tika(user: AdminWriter, cfg: Cfg) -> TikaTestResult:
"""Check that a Tika server answers at documents.tika_url, and say which version it is."""
t0 = time.time()
try:
said = tika.version(cfg)
except ValueError as e:
return TikaTestResult(ok=False, error=str(e))
return TikaTestResult(ok=True, version=said, ms=int((time.time() - t0) * 1000))


@router.get("/settings/decisions/status")
def decision_status(user: AdminReader, cfg: Cfg) -> DecisionStatus:
"""Who takes routine decisions, and whether a local Laya model can here: on this machine (Apple Silicon, once
Expand Down
43 changes: 31 additions & 12 deletions fastapi_backend/app/domain/convert.py
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@
from email.utils import getaddresses, parsedate_to_datetime
from html.parser import HTMLParser

from . import anytopdf, netguard, store
from . import anytopdf, netguard, store, tika

R = store.R
OFFICE = {
Expand Down Expand Up @@ -187,19 +187,36 @@ def _has_msg():
return True


def _can(cfg):
"""What this server can make into PDFs itself, or with anytopdf: {office, pages, msg}."""
by = None if anytopdf.mode(cfg) == "lens" else anytopdf.available(cfg)
lo, web = bool(soffice(cfg)) or fetchable("libreoffice"), web_capture(cfg)
office, pages = lo or by == "node", bool(web or lo or by)
return {"office": office, "pages": pages, "msg": pages and _has_msg()}


def capabilities(cfg):
"""What this server can make into PDFs: Office files (LibreOffice), text, Markdown, saved web pages and emails
(Chromium or LibreOffice), Outlook .msg emails (those, and the extract-msg package), and web pages captured from
their address (Chromium). anytopdf does the same where Lens can't (anytopdf.py): emails, text and pages here, and
Office files too on a conversion node."""
by = None if anytopdf.mode(cfg) == "lens" else anytopdf.available(cfg)
lo, web = bool(soffice(cfg)) or fetchable("libreoffice"), web_capture(cfg)
office, pages = lo or by == "node", bool(web or lo or by)
return {"office": office, "pages": pages, "msg": pages and _has_msg(), "web": web}
Office files too on a conversion node. A Tika server reads all of those as text where nothing here can (tika.py)."""
got = {**_can(cfg), "web": web_capture(cfg), "tika": tika.reads_text(cfg)}
if got["tika"]:
got.update(office=True, pages=True, msg=True)
return got


def unavailable(cfg, name):
"""Why this server can't read this document (None when it can): what it would need."""
"""Why this server can't read this document (None when it can): what it would need. A Tika server reads, as text,
any document Lens can't make into a PDF here."""
why = unconvertible(cfg, name)
if why and tika.reads_text(cfg) and ext_of(name) in store.DOCUMENT_EXT:
return None
return why


def unconvertible(cfg, name):
"""Why this server can't make this document into a PDF (None when it can): what it would need."""
ext = ext_of(name)
if ext in store.CAMERA_EXT:
if anytopdf.mode(cfg) == "lens" or not anytopdf.available(cfg):
Expand All @@ -208,11 +225,13 @@ def unavailable(cfg, name):
return None
if ext not in store.DOCUMENT_EXT or ext == ".pdf":
return None
can = capabilities(cfg)
can = _can(cfg)
if ext in OFFICE and not can["office"]:
return f"converting {word(name)}s needs LibreOffice on the server (the lens:full image) or an anytopdf conversion node"
return (
f"converting {word(name)}s needs LibreOffice on the server (the lens:full image), an anytopdf conversion node or a Tika server"
)
if ext not in OFFICE and not can["pages"]:
return f"converting {word(name)}s needs Chromium or LibreOffice on the server (the lens:full image)"
return f"converting {word(name)}s needs Chromium or LibreOffice on the server (the lens:full image), or a Tika server"
if ext == ".msg" and not can["msg"]:
return "reading Outlook .msg emails needs the extract-msg package on the server (pip install -e '.[msg]')"
return None
Expand Down Expand Up @@ -270,7 +289,7 @@ def office_pdf(cfg, src, out):
"""An Office, OpenDocument or RTF file made into a PDF by LibreOffice, offline and with a profile of its own."""
exe = soffice(cfg)
if not exe:
raise Unavailable(unavailable(cfg, src) or "LibreOffice isn't installed")
raise Unavailable(unconvertible(cfg, src) or "LibreOffice isn't installed")
with tempfile.TemporaryDirectory(prefix="lens-office-") as tmp:
doc = pathlib.Path(tmp) / f"document{ext_of(src)}"
shutil.copyfile(src, doc)
Expand Down Expand Up @@ -827,7 +846,7 @@ def to_pdf(cfg, src, out, say=None):
"""Make `out`, the PDF of the document at `src`. Returns what was learnt on the way: {by, title?, email?,
attachments?}. Unavailable when the server can't; ValueError when the file can't be read. A program installed
the first time it's needed (fetchable) is installed first, which `say` tells."""
why = unavailable(cfg, src)
why = unconvertible(cfg, src)
if why:
raise Unavailable(why)
ext, path, via = ext_of(src), pathlib.Path(src), by_anytopdf(cfg, src)
Expand Down
Loading
Loading