Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,12 @@ The backend (`fastapi_backend`) and the frontend (`nextjs-frontend`) are version

## Unreleased

- **Dates read as days, for agents.** With Settings → Analysis → Read dates on (off by default), analysis reads the
dates each line gives as days on the calendar: "next Tuesday" in a recording made on a Wednesday is the Tuesday
after, "last week" Monday to Sunday before, "March 2024" all of March, worked out from the asset's own date. Agents
ask what was said about a day or a range with `GET /api/v1/dates`, the MCP tool `list_dates` and the assistant's
`find_dates`, each date with its line and a link to that moment. English rules, no model; a weekday alone or
"10/11/2026" is left out rather than guessed ([Dates](docs/processing.md#dates)).
- **Choose the tools, a limit and the namespaces of an answer.** A conversation, or one question, can pick the tool
groups or tools its answers may use (`tools`), cap their tool calls (`max_tool_calls`, up to the new Settings → AI
assistant → Most tool calls a question may ask for, 50 by default) and narrow one answer to its own namespaces
Expand Down
11 changes: 11 additions & 0 deletions DEVLOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,17 @@

Plans and progress for work in flight. Newest first.

## 2026-10-17 · Dates read as days

Adeel: dates extraction. `dates.py` reads the dates each line gives (written out, numeric, and relative to the day)
into `date_said` as first and last days, worked out from the asset's `recorded_at`, when Settings → Analysis → Read
dates is on. `GET /api/v1/dates`, MCP `list_dates` and the assistant's `find_dates` find them by day or range with
their lines. English rules in Lens's own code rather than dateparser: dateparser couldn't read "next Tuesday" or "this
Friday", and its own search over a line finds dates in ordinary words.

Refine later: DATE entities named by their day, so "next Tuesday" and "20 October" in two assets meet in the graph;
other languages' month and day names; a Dates view in the web app (a calendar of what was said); times of day.

## 2026-10-17 · Entities found by spaCy

Adeel: NLP libraries for named entities. Settings → Analysis → Entity extraction → spaCy existed but needed spaCy
Expand Down
14 changes: 10 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,9 +62,10 @@ measured numbers ([requirements and their checks](docs/requirements.md)). Work t
listed under [In progress](#in-progress), and what comes next under [Planned](#planned).

- **Connect your agents.** Lens is an MCP server (`/mcp`): agents search, find assets by face, voice, entity, topic,
type, content type or date in one call, classify an asset or a text with the decision model, read assets' text and
notes, cite exact moments, find similar content, explore the graph with Cypher or SPARQL or browse it as folders and
files, suggest topics and graph changes, edit and merge entities and topics (with undo), write notes and make
type, content type or date in one call, find what was said about a day, classify an asset or a text with the
decision model, read assets' text and notes, cite exact moments, find similar content, explore the graph with Cypher
or SPARQL or browse it as folders and files, suggest topics and graph changes, edit and merge entities and topics
(with undo), write notes and make
podcasts. They sign in with OAuth and see only what their person may read; the account menu copies the MCP address in
one click. A namespace can also have an MCP server of its own (`/mcp/<namespace>`) and answer OpenAI's chat
completions API for any app, cited from that namespace alone, each off until its owner turns it on, with API keys that
Expand Down Expand Up @@ -121,10 +122,15 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p
rules, without a model. Find what is most like a passage, an asset, a topic or an entity, in a Similar
tab on every asset or by selecting text anywhere, and, with anytopdf, photos by what they show. Pages and shots are
described in words by the model provider's vision model or, off by default, by Florence-2 on your own server.
Transcripts can be cleaned up before analysis (stutters, "um"s, spacing and, with a local model, punctuation put
back), entities found by spaCy instead of the rules, and the dates people say read as days on the calendar, each off
by default.
([Graph](docs/graph.md), [entity pages](docs/notes.md#entity-pages),
[similar content](docs/processing.md#similar-content), [graph history](docs/graph-history.md),
[topics](docs/topics.md), [video](docs/video.md), [NSFW classification](docs/nsfw.md),
[long recordings](docs/speech-providers.md#long-recordings); [checks](docs/requirements.md#understand-it))
[long recordings](docs/speech-providers.md#long-recordings),
[cleaning up transcripts](docs/processing.md#cleaning-up-transcripts), [dates](docs/processing.md#dates);
[checks](docs/requirements.md#understand-it))
- **Learn from it.** Make a two-host podcast episode about what you pick, every claim cited and checked against its
source, read aloud in two voices. Keep notes beside every asset, entity and topic, written by you or the assistant, in
a tree where notes are renamed, moved, copied and pinned (pins are your own) and each note shows the notes it's
Expand Down
12 changes: 12 additions & 0 deletions docs/api.md
Original file line number Diff line number Diff line change
Expand Up @@ -1258,6 +1258,18 @@ An entity's `page` is its page in the web app, by name: `/wiki/<namespace>/<Name
spaces), or a name it had before a rename or a merge. It answers `{kind, id, namespace, title, path, canonical}`;
`canonical` is false when the address is another way of writing `path` ([Notes](notes.md#entity-pages)).

## dates

```
GET /api/v1/dates
```

The dates assets' lines give, as days on the calendar ([Dates](processing.md#dates)): `start` and `end`
(`YYYY-MM-DD`, or `YYYY-MM` and `YYYY` for their first and last day) keep those that fall between them, `ns`, `recording`
and `q` narrow them. Each item has `day` and `until` (its first and last day), `period` (`day`, `weekend`, `week`,
`month`, `year`), `said` (the words), `relative` (worked out from the asset's own date), and the line: `recording_id`,
`title`, `idx`, `t0`, `time`, `speaker` and `text`. `reading` says whether analysis reads dates now.

## metadata

```
Expand Down
3 changes: 2 additions & 1 deletion docs/assistant.md
Original file line number Diff line number Diff line change
Expand Up @@ -138,7 +138,8 @@ engine), `POST /voice/transcribe` (a clip as the raw body; nothing is kept), `PO
When the configured model supports function calling, chat becomes an agent.

- **What it can look things up with:** assets' text (transcripts and documents, and on-screen text), assets, passages,
outputs, entities and their mentions and timelines, the knowledge graph, people, and content similar to a text,
outputs, entities and their mentions and timelines, dates said as days on the calendar (`find_dates`), the
knowledge graph, people, and content similar to a text,
an asset, a topic or an entity (`find_similar`), and what happened in Lens, newest first (`timeline`, the
[stream of activities](automations.md#timeline-reads); `unreviewed` for what the person hasn't marked reviewed),
and assets by any attribute: faces seen, people heard, entities, topics, kind, content type, dates and the rest,
Expand Down
1 change: 1 addition & 0 deletions docs/mcp.md
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,7 @@ For an app that speaks OpenAI's API rather than MCP, the namespace can also answ
| `cite` | a citation for a moment (a line or a second, and a few lines from there): the words, who said or wrote them, the asset and date, the time, the link, and all of it as Markdown |
| `list_entities` | people, organisations, products, places, events, works and terms mentioned, by name, type, namespace or asset |
| `get_entity` | one entity: its names, its page's link, how often and when it is mentioned, what it is mentioned with, who mentions it most, and the lines that mention it |
| `list_dates` | what was said about a day or a range: dates said in assets as days on the calendar ([Dates](processing.md#dates)), each with the words, the line and a link to that moment |
| `explore_graph` | the knowledge graph around an entity (`e12`) or a person (`s3`), one or two steps out |
| `find_path` | the shortest chain of links between two of them, with lines that show each link |
| `sparql` | a read-only SPARQL query (SELECT, ASK, CONSTRUCT or DESCRIBE) over one namespace's linked data ([RDF](rdf.md)): JSON results, or Turtle |
Expand Down
19 changes: 19 additions & 0 deletions docs/processing.md
Original file line number Diff line number Diff line change
Expand Up @@ -222,6 +222,25 @@ retype, describe, merge and hide it.
- **Merge suggestions:** same letters (ignoring case, spaces and punctuation), acronyms, one name containing the other,
close spellings, and names that sound alike (likely transcription errors).

### Dates

With `analysis.dates` on (Settings → Analysis → Read dates; off by default), analysis reads each line for the dates it
gives and keeps them as days on the calendar, so agents and the API can ask what was said about a day, a month or a
range: `GET /api/v1/dates?start=2026-10&end=2026-10`, the MCP tool `list_dates`, the assistant's `find_dates`. Each
date keeps the words said, its first and last day and how long it is (a day, a weekend, a week, a month or a year), and
its line, linked to that moment.

- **Said relative to the day:** "yesterday", "tomorrow", "in three days", "two weeks ago", "next month", "this
weekend" are worked out from the asset's own date (its `recorded_at`), and marked `relative`. "this Friday" is the
coming one, "next Friday" the Friday of the week after, "last Friday" the one before. "last week" is Monday to Sunday.
- **Written out:** "3 March 2024", "March 3rd", "March 2024" (the whole month), "2026-10-20", "20/10/2026", and a year
after a word that makes it one ("in 2024", "since 1998"). A day without its year takes the asset's year.
- **Left out rather than guessed:** a weekday on its own ("on Friday": the one before or after?), a date that reads two
ways ("10/11/2026"), "may" and short month names in small letters ("I may", "to mar"), and a bare number like "2000
people". Without an asset date, only dates that give their year are kept.
- The rules read English. Nothing is sent anywhere and no model is called. Turning it on reads dates as assets are
analysed; Reprocess (Analyze) reads them for assets already there.

### Entity setup

Each namespace has an entity setup (Entities → Setup, `GET`/`PUT /api/v1/namespaces/{name}/entity-setup`): the types
Expand Down
5 changes: 5 additions & 0 deletions docs/requirements.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,7 @@ request, to another check that proves the same thing; loosening a requirement or
| An MCP server at `/mcp`: agents search, read assets' text and notes, and cite exact moments | [test_mcp.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_mcp.py) `test_search_and_cite`, `test_reading_recordings`, `test_note_tools` |
| Agents find assets by face, voice, entity, topic, type, content type and date in one call, each with the moments that match | [test_find.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_find.py) `test_by_face_voice_entity_and_date`, `test_by_topic_type_content_type_and_text`, `test_find_for_agents` |
| Agents classify an asset or a text against labels, content types or topics with the decision model, acting only on a sure answer | [test_classify.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_classify.py) `test_labels_given_and_what_the_model_reads`, `test_vocabularies_and_applying_a_sure_answer`, `test_agents_classify` |
| Agents find what was said about a day or a range, each date with its line and a link to that moment | [test_dates.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_dates.py) `test_agents_find_what_was_said_about_a_day` |
| Agents find similar content | [test_mcp.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_mcp.py) `test_search_and_cite`; [test_similar.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_similar.py) `test_like_a_text_by_its_key_words`, `test_like_a_resource_by_words_topics_and_entities` |
| Agents explore the graph with Cypher or SPARQL, or browse it as folders and files | [test_mcp.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_mcp.py) `test_graph_tools`, `test_sparql`; [test_file_tree.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_file_tree.py) `test_agents_browse_it_over_mcp` |
| Agents suggest topics and graph changes, and edit and merge entities and topics | [test_mcp.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_mcp.py) `test_topic_tools`, `test_editing_topics_and_entities` |
Expand Down Expand Up @@ -143,6 +144,10 @@ request, to another check that proves the same thing; loosening a requirement or
| What is most like a passage, an asset, a topic or an entity | [test_similar.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_similar.py) `test_like_a_text_by_its_key_words`, `test_like_a_resource_by_words_topics_and_entities`, `test_like_a_topic_and_an_entity`; [similar.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/similar.test.tsx) |
| With anytopdf, photos found by what they show | [test_photos.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_photos.py) `test_photos_found_by_what_they_show`, `test_off_by_default` |
| Pages and shots described by the provider's vision model or, off by default, by Florence-2 on this server, fetched pinned and checked when picked | [test_florence.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_florence.py) `test_florence_describes_pictures_without_a_model_provider`, `test_florence_is_fetched_pinned_and_checked`, `test_admins_and_agents_choose_what_describes_pictures`, `test_a_picture_is_read_as_the_model_was_trained` |
| Transcripts cleaned up before analysis (stutters, "um"s, spacing), off by default, each change in the edit history with Revert and people's corrections left alone | [test_cleanup.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_cleanup.py) `test_off_by_default_changes_nothing`, `test_clean_step_tidies_lines_and_keeps_them_in_the_history`, `test_people_s_corrections_outrank_the_rules` |
| Punctuation put back in transcripts that lack it by a local model, fetched only when picked | [test_cleanup.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_cleanup.py) `test_punctuation_put_back_where_it_is_missing`, `test_without_the_model_the_rules_run_alone`, `test_the_model_is_fetched_only_when_it_is_asked_for` |
| Entities found by spaCy, fetched the first time it's picked, with the defined entities first and the rules until it's there | [test_spacy_entities.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_spacy_entities.py) `test_defined_entities_come_first_then_spacy_s_names`, `test_the_analyze_step_and_workflows_use_it`, `test_without_spacy_the_rules_find_them`, `test_spacy_is_fetched_only_when_it_is_picked` |
| Dates said in an asset read as days ("next Tuesday", "last week"), off by default, with dates that read two ways left out | [test_dates.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_dates.py) `test_dates_are_read_as_days`, `test_what_could_be_two_dates_is_left_out`, `test_off_by_default_reads_none` |

## Learn from it

Expand Down
72 changes: 72 additions & 0 deletions fastapi_backend/app/api/mcp_tools.py
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@
from app.api.v1.routes import activities as activity_routes
from app.api.v1.routes import automation_vars as var_routes
from app.api.v1.routes import automations as automation_routes
from app.api.v1.routes import dates as date_routes
from app.api.v1.routes import entities as entity_routes
from app.api.v1.routes import graph as graph_routes
from app.api.v1.routes import namespaces as namespace_routes
Expand Down Expand Up @@ -1197,6 +1198,77 @@ def _scope(namespace: str | None) -> str:
return f"ns:{namespace}" if namespace else "all"


@tool(
"list_dates",
"Find what was said about a day",
'Dates said in assets, read as days on the calendar: "next Tuesday" in a recording made on a Wednesday is the '
'Tuesday after, "last week" the seven days before that Monday, "March 2024" all of March. Give start and end '
"(YYYY-MM-DD, or YYYY-MM or YYYY) for what falls between them, the earliest first, each with the words said, the "
"line and a url to that moment. relative: worked out from the asset's own date. Only assets analysed while Settings "
"→ Analysis → Read dates was on have them (reading says whether it is).",
Arg("start", "string", "from this day on: YYYY-MM-DD, YYYY-MM or YYYY", max_length=10),
Arg("end", "string", "up to this day: YYYY-MM-DD, YYYY-MM or YYYY", max_length=10),
Arg("namespace", "string", "only this namespace"),
Arg("recording_id", "integer", "only this asset"),
Arg("query", "string", "only dates whose words or line have this text", max_length=200),
Arg("limit", "integer", "how many", default=25, minimum=1, maximum=100),
Arg("offset", "integer", "skip this many (for the next page)", default=0, minimum=0, maximum=10000),
)
def list_dates(
ctx: Context,
start: str | None,
end: str | None,
namespace: str | None,
recording_id: int | None,
query: str | None,
limit: int,
offset: int,
) -> dict[str, Any]:
res = date_routes.list_dates(
ctx.acl,
ctx.user,
ctx.db,
ctx.cfg,
start=start or "",
end=end or "",
ns=namespace,
recording=recording_id,
q=query or "",
limit=limit,
offset=offset,
)
out: dict[str, Any] = {
"reading": res.reading,
"total": res.total,
"dates": [
{
k: v
for k, v in {
"day": d.day,
"until": d.until if d.until != d.day else None,
"period": d.period,
"said": d.said,
"relative": d.relative or None,
"recording_id": d.recording_id,
"title": d.title,
"namespace": d.namespace,
"recorded_at": d.recorded_at,
"line": d.idx,
"at": d.time,
"speaker": d.speaker,
"text": _cut(d.text),
"url": ctx.link(d.recording_id, d.t0),
}.items()
if v is not None
}
for d in res.items
],
}
if offset + len(res.items) < res.total:
out["next_offset"] = offset + len(res.items)
return out


@tool(
"explore_graph",
"Explore the knowledge graph",
Expand Down
2 changes: 2 additions & 0 deletions fastapi_backend/app/api/v1/router.py
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@
components,
content_types,
custom_nodes,
dates,
entities,
entity_setup,
extensions,
Expand Down Expand Up @@ -114,6 +115,7 @@
searches,
speakers,
entities,
dates,
topics,
graph,
graph_history,
Expand Down
Loading
Loading