Skip to content

feat(chat): each answer records what produced it - #469

Merged
adeelahmad merged 8 commits into
mainfrom
claude/project-thread-r960xc-c1-record
Oct 17, 2026
Merged

adeelahmad merged 8 commits into
mainfrom
claude/project-thread-r960xc-c1-record

Conversation

@adeelahmad

Copy link
Copy Markdown
Owner

Requested by Adeel · project thread

Description

Before: an answer showed only the model that wrote it. Nothing said which tools it was offered, what scope it answered in, how many rounds it used, what it cost or how it ended, and ledger rows named the conversation but not the answer.

After: every answer keeps a record (meta): the model and the host name of its server (never the address or key), the tools offered and asked for, the scope after auto-scope, its limits and use, how it ended (answered, direct, limit, fallback, stopped, error), tool results that read like instructions to an AI, how long it took, Lens's version, and tokens and cost from the ledger rows that name it. Questions keep how they were sent (web, api, voice, room, automation) and their options. An i in the answer's header opens Details (a sheet on a phone) with all of it, the calls made for it and the conversation's cost this month; old answers say they weren't recorded. Each tool step shows how long it took and whether it failed.

How: the answer's id is taken right after the question is saved, and the answer runs in activity.scope(chat:<cid>, chat_message:<mid>) in the web chat, chat rooms and the automations' agent. A streamed answer can't hold a with scope across its yields (each step runs in a fresh copy of the request's context in the threadpool), so activity.scoped() enters it per step. The stream sends meta just before done; GET /api/v1/chats/{cid}/messages/{mid} returns one message with its record and ledger rows. App completions keep no messages, so their record comes back in lens.record, without tokens and cost.

Motivation and Context

Module C1 of the chat-gaps plan: record all data about a response (tools, model, scope), so people can trust and debug answers and agents can read them.

Screenshots (if appropriate):

n/a (jest renders the popover and the phone sheet).

Steps to reproduce (if appropriate):

Ask anything in Chat, then press the i next to the answer's model.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change; ! in the title)

Checklist:

  • The title is a Conventional Commit.
  • My code follows the code style of this project (ruff, ruff format, mypy, eslint, prettier, tsc clean).
  • I added or updated tests (tests/api/test_chat_records.py; records in rooms, the agent, completions and flagged tools; __tests__/chat-details.test.tsx).
  • My change requires documentation updates, and I have updated docs/ accordingly (assistant.md › What each answer records, api.md, database.md › Upgrades).
  • Users need more than the title to understand it, so I wrote it under ## Unreleased in CHANGELOG.md.

Done questions (AGENTS.md)

  • Goal: every answer records what produced it, shown in the web app and in the API; merged on green CI.
  • Agents: GET /chats/{cid} and GET /chats/{cid}/messages/{mid} return the record and ledger rows; completions return lens.record. No assistant tool: the assistant can't usefully read its own record. MCP read_chat comes with C3.
  • Zero setup: nothing to set up.
  • Fewest clicks and keystrokes: one click (or Enter on the focused i).
  • Configurable: n/a, no new setting.
  • Light: no new dependency; one ledger total per answer.
  • Private and secure: owner-only endpoint and chat_message:<id> activity; the record names the model server's host name only; threat model outputs table updated.
  • Cost: no model calls; every model call made for an answer names it in the ledger; costs labelled estimates.
  • User decides: defaults unchanged; nothing acts.
  • Upgrades: no step needed: old messages have no meta and readers show what's there (docs/database.md › Upgrades).
  • Honest: the cost is labelled an estimate from the prices in Settings; old answers say they weren't recorded.
  • Tests, docs and generated files updated (make openapi).
  • Open standards; no new dependencies.
  • Result reported with evidence; refine later: first-token latency, the system prompt's version, model parameters once settable, a Costs page link once it shows one resource, tokens for completions.

Local checks: backend pytest -n 4 1332 passed, 20 skipped; ruff, ruff format and mypy clean; frontend jest 139 suites / 962 tests passed, tsc and eslint clean; mkdocs build --strict clean; ci/validate_threat_model.py agrees; OpenAPI regenerated and matches.


Generated by Claude Code

Importing app.domain.ai_tools on its own failed: it imported automations,
which loads the actions, and the agent's action subclasses Toolbox before
ai_tools had defined it. Running one test file (tests/api/test_chats.py)
hit it. ai_tools now imports automations last.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HXPWXcSCQb9hWPr7xkZu7B
Every answer keeps `meta`: the model and the host name of its server,
the tools offered and asked for, the scope it answered in (after
auto-scope), its limits and what it used, how it ended (answered,
direct, limit, fallback, stopped, error), tool results that read like
instructions to an AI, how long it took, Lens's version, and tokens and
cost from the ledger rows that name it. Questions keep how they were
sent (web, api, voice, room, automation) and their options.

The answer's id is taken once the question is saved and the answer
runs in activity.scope(chat:<cid>, chat_message:<mid>), in the web
chat, chat rooms and the automations' agent; the streamed answer holds
the scope across its steps (activity.scoped). App completions keep no
messages, so their record comes back in lens.record. The stream sends
`meta` before `done`; GET /chats/{cid}/messages/{mid} returns one
message with its record and ledger rows. Each tool step keeps its time
and whether it failed. The web app shows it all from an i in the
answer's header (a sheet on a phone); old answers say they weren't
recorded.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HXPWXcSCQb9hWPr7xkZu7B
@adeelahmad adeelahmad self-assigned this Oct 17, 2026
The last check asked for 2026-10-17's note after the call without a date had made today's note, so on that day it
found the note the deleted root had left behind. It now asks for a day no earlier call makes.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HXPWXcSCQb9hWPr7xkZu7B

Copy link
Copy Markdown
Owner Author

I ported the fix from #470 here in acf97d7. Without it, tests/api/test_notebook.py::test_daily_notes fails on every branch all day on 2026-10-17, because the test depends on the date and doesn't come from this PR. The fix does nothing once main has it.


Generated by Claude Code

The answer's record uses C2's reserved answer_id for its ledger scope and is saved under the question (parent);
a message of any version can be read with its record and ledger rows.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HXPWXcSCQb9hWPr7xkZu7B
Keeps C2's versions UI (regenerate, parent, VersionSwitch) with the answer's Details and record; a question
said in voice mode still sends via. useIsNarrow lays out wide where there's no matchMedia, as Details now
renders in every saved answer's header.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HXPWXcSCQb9hWPr7xkZu7B
Keeps the sandbox import and web citations (web pages stay; recordings follow access) with get_message's
readable-citation helper; prose_version is the sha256 of the merged threat-model.md.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HXPWXcSCQb9hWPr7xkZu7B
@adeelahmad
adeelahmad merged commit e3c9d66 into main Oct 17, 2026
11 checks passed
@adeelahmad
adeelahmad deleted the claude/project-thread-r960xc-c1-record branch October 17, 2026 02:24
adeelahmad pushed a commit that referenced this pull request Oct 17, 2026
#469 merged while this was open: its line moves from In progress to the built list, with its checks. The
scorecard (#487) is merged too, so Planned keeps only what it doesn't measure yet.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MTK8aqzLBhdhfCmVf2LaWZ
adeelahmad pushed a commit that referenced this pull request Oct 17, 2026
C1 (#469), web search (#472) and the sandbox (#473) are in main. The
routes keep C5's web pages in citations alongside the export, and the
export test now checks a cited web page stays through access changes.
The OpenAPI schema and client are regenerated with make openapi's two
commands, and the threat model's prose_version is recomputed.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HXPWXcSCQb9hWPr7xkZu7B
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants