Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,9 @@ The backend (`fastapi_backend`) and the frontend (`nextjs-frontend`) are version
how long it took, how it ended and which tool results read like instructions to an AI, with the calls made for it.
Agents read the same record over the API (`GET /api/v1/chats/<id>/messages/<mid>`), and each tool step shows how long
it took. Answers from before have no record ([What each answer records](docs/assistant.md#what-each-answer-records)).
- **Download notes as Markdown.** A note's ⋯ menu in the tree downloads it as a Markdown file, and the Notes page's ⋯
menu downloads all of a namespace's notes as a zip in folders that follow the tree, for Obsidian and other notes apps
([Downloading as Markdown](docs/notes.md#downloading-as-markdown)).
- **Automations for agents.** Agents list, save, try and run automations through the MCP tools `automations`,
`automation_actions`, `save_automation`, `test_automation` and `run_automation`, with problems in plain words. The
assistant saves them switched off and asks before switching one on or running it
Expand Down
5 changes: 5 additions & 0 deletions ci/clicks/journeys.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -651,6 +651,11 @@ export const JOURNEYS = [
from: "/",
async run(ctx) {
const { page, meter } = ctx;
// A slow machine can still be reading import-file's pi-episode here, and its mentions of Lens and Raspberry Pi
// landing between the merge and its Undo leave both out of the graph for graph-merge; so wait for the jobs to
// finish (not counted)
const idle = page.getByRole("button", { name: /^Activity: (Nothing running|\d+ failed)$/ });
await madeInBackground(ctx, "/", idle, "/", "the imports' jobs finished");
await go(ctx, "Entities");
const main = page.getByRole("main");
await meter.click(main.getByRole("checkbox", { name: "Pick Lens", exact: true }), "tick Lens");
Expand Down
5 changes: 4 additions & 1 deletion docs/api.md
Original file line number Diff line number Diff line change
Expand Up @@ -818,6 +818,7 @@ GET /api/v1/notes
POST /api/v1/notes
POST /api/v1/notes/today
GET /api/v1/notes/search
GET /api/v1/notes/export
GET /api/v1/notes/deleted
POST /api/v1/notes/deleted/{pid}/restore
DELETE /api/v1/notes/deleted/{pid}
Expand All @@ -827,6 +828,7 @@ DELETE /api/v1/notes/{pid}
POST /api/v1/notes/{pid}/move
PUT /api/v1/notes/{pid}/pin
DELETE /api/v1/notes/{pid}/pin
GET /api/v1/notes/{pid}/markdown
GET /api/v1/notes/{pid}/history
GET /api/v1/notes/{pid}/history/{vid}
POST /api/v1/notes/{pid}/history/{vid}/restore
Expand All @@ -843,7 +845,8 @@ matches first, each with the words around the match ([Finding notes](notes.md#fi
moves a page to recently deleted for 30 days: `deleted` lists them, `restore` puts one back where it was and
`DELETE /notes/deleted/{pid}` deletes one for good ([Deleting](notes.md#deleting-and-putting-back)). `move` puts a
free note inside `parent` before `before`. `PUT …/pin` pins a page for you (`pinned` on each page says whether you
did) and `DELETE …/pin` unpins it ([Pinned notes](notes.md#pinned-notes)). `history` lists what the page was before
did) and `DELETE …/pin` unpins it ([Pinned notes](notes.md#pinned-notes)). `markdown` gives a page as a Markdown file and
`export?ns=` every page of a namespace as a zip of them ([Downloading](notes.md#downloading-as-markdown)). `history` lists what the page was before
each change, and `restore` puts an earlier version back (what it was becomes a version too). `files`
lists the files and images attached to a page; `blobs` uploads (in chunks) and reads them, and `DELETE …/files?key=`
removes one.
Expand Down
15 changes: 12 additions & 3 deletions docs/notes.md
Original file line number Diff line number Diff line change
Expand Up @@ -128,9 +128,9 @@ Free notes sit in a tree in the left navigation, under the main links. The links
long tree has the whole height. A row opens its note, its arrow shows the notes inside it (the tree remembers which are
open) and **+** adds a note inside. Each row's **⋯** menu, which always shows on a touch screen, adds a note inside,
renames the note in place (Enter saves, Escape leaves it as it was), moves it (**Move to…** finds the note to put it in
by name, Enter picks the first; **Move to the top**), pins it ([below](#pinned-notes)) and deletes it with Undo.
Dragging a note onto another puts it inside, and onto the Notes heading moves it to the top. The pages of recordings
and entities aren't in the tree: they open from those things.
by name, Enter picks the first; **Move to the top**), pins it ([below](#pinned-notes)), downloads it as Markdown and
deletes it with Undo. Dragging a note onto another puts it inside, and onto the Notes heading moves it to the top. The
pages of recordings and entities aren't in the tree: they open from those things.

`POST /api/v1/notes/{pid}/move` with `{parent}` moves a note (`null` for the top); `PATCH /api/v1/notes/{pid}` with
`{title}` renames it.
Expand All @@ -151,6 +151,15 @@ read access; up to 50 per namespace. **Unpin** (in the menu, or the button on it
`PUT /api/v1/notes/{pid}/pin` pins and `DELETE` unpins; each page in `GET /api/v1/notes` says whether you pinned it
(`pinned`).

## Downloading as Markdown

**Download as Markdown** in a note's ⋯ menu in the tree saves it as a `.md` file, and **Download all as Markdown**
in the Notes page's ⋯ menu saves every page of the namespace as a zip of them, for Obsidian and other notes apps. Each
file starts with front matter (its title, date, where it's filed, its summary, what it's the page of), then its text:
links to other notes become `[[Title]]` wiki links and the rest the names they link to. In the zip, the notes inside a
note sit in a folder named like it, and the pages of recordings, entities and topics under `Pages of things`.
`GET /api/v1/notes/{pid}/markdown` and `GET /api/v1/notes/export?ns=` give the same files to agents.

## Deleting and putting back

**Delete** on a page deletes it at once, with no question first: the notice has **Undo**, which puts it back and opens
Expand Down
49 changes: 48 additions & 1 deletion fastapi_backend/app/api/v1/routes/notebook.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,11 +7,12 @@
from __future__ import annotations

import contextlib
import tempfile
from typing import Any
from urllib.parse import quote

from fastapi import APIRouter, HTTPException, Query, Request
from fastapi.responses import StreamingResponse
from fastapi.responses import Response, StreamingResponse
from starlette.concurrency import run_in_threadpool

from app.api.deps import Access, Acl, Cfg, CurrentUser, Db, Principal, Writer, domain_errors
Expand Down Expand Up @@ -170,6 +171,35 @@ def _trashed(acl: Access, db: DB, pid: int, role: str = "viewer") -> dict[str, A
return t


def _attachment(name: str) -> str:
ascii_name = name.encode("ascii", "replace").decode().replace('"', "").replace("?", "_")
return f"attachment; filename=\"{ascii_name}\"; filename*=UTF-8''{quote(name)}"


@router.get(
"/export",
response_class=StreamingResponse,
responses={200: {"content": {"application/zip": {}}, "description": "a zip of Markdown files"}},
)
def export_notes(ns: str, user: CurrentUser, acl: Acl, db: Db) -> StreamingResponse:
"""Every page of the namespace as Markdown files in a zip, for other notes apps (Obsidian and the like): free notes
in folders that follow the tree, the pages of things under "Pages of things"."""
sid = acl.namespace(ns)
out = tempfile.SpooledTemporaryFile(max_size=32 * 1024 * 1024)
notebook.export(db, sid, out)
out.seek(0)

def body():
try:
while part := out.read(256 * 1024):
yield part
finally:
out.close()

name = f"{notebook.file_name(ns)} notes.zip"
return StreamingResponse(body(), media_type="application/zip", headers={"Content-Disposition": _attachment(name)})


@router.get("/deleted")
def deleted_pages(
user: CurrentUser,
Expand Down Expand Up @@ -278,6 +308,23 @@ def _version_out(db: DB, v: dict[str, Any], cls: type[NoteVersionItem] = NoteVer
return cls(**{**{k: x for k, x in v.items() if k not in ("by", "page", "doc")}, "by": (who or {}).get("email")})


@router.get(
"/{pid}/markdown",
response_class=Response,
responses={200: {"content": {"text/markdown": {}}, "description": "the page as a Markdown file"}},
)
def page_markdown(pid: int, user: CurrentUser, acl: Acl, db: Db) -> Response:
"""The page as a Markdown file of its own: front matter with its title, date, filing and summary, then its text,
with links to other pages as [[Title]] wiki links."""
p = _page(acl, db, pid)
titles = {r["id"]: r["title"] for r in notebook.tree(db, p["space"], everything=True)}
return Response(
notebook.markdown(p, titles),
media_type="text/markdown; charset=utf-8",
headers={"Content-Disposition": _attachment(f"{notebook.file_name(p['title'])}.md")},
)


@router.get("/{pid}/history")
def page_history(pid: int, user: CurrentUser, acl: Acl, db: Db, limit: int = Query(50, ge=1, le=100)) -> NoteHistory:
"""What the page was before each change to its title, summary or text, newest first."""
Expand Down
62 changes: 62 additions & 0 deletions fastapi_backend/app/domain/notebook.py
Original file line number Diff line number Diff line change
Expand Up @@ -27,8 +27,10 @@
import base64
import datetime as dt
import hashlib
import json
import os
import re
import zipfile

from . import activity, store

Expand Down Expand Up @@ -810,6 +812,66 @@ def labels(db, targets):
return out


EXPORT_NAME_MAX = 100 # characters of a file's name, from the page's title
_UNSAFE = re.compile(r'[\\/:*?"<>|\x00-\x1f]+')


def file_name(title):
"""A page's title as a file name that works everywhere: no slashes or other marks file systems refuse."""
name = _UNSAFE.sub(" ", str(title or "")).strip(" .")
name = " ".join(name.split())[:EXPORT_NAME_MAX].rstrip(" .")
return name or "Untitled"


def markdown(p, titles=None):
"""A page as a Markdown file of its own, for other notes apps: YAML front matter (its title, date, where it's
filed, its summary, what it's the page of), then its text. Links to other pages become [[Title]] wiki links (their
title now, from `titles` by id, else as written); links to anything else become its name."""
titles = titles or {}

def link(m):
sign, label, kind, key = m.groups()
if kind == "page":
return f"[[{titles.get(int(key), label)}]]"
return label if sign == "@" else f"#{label}"

head = {"title": p.get("title"), "date": p.get("date"), "place": p.get("place"), "summary": p.get("summary")}
head["page_of"] = p.get("about")
lines = ["---", *(f"{k}: {json.dumps(v, ensure_ascii=False)}" for k, v in head.items() if v), "---", ""]
return "\n".join(lines) + MENTION.sub(link, p.get("body") or "").strip() + "\n"


THINGS = {"recording": "Recordings", "entity": "Entities", "topic": "Topics", "collection": "Collections", "speaker": "Speakers"}


def export(db, sid, out):
"""Every page of the namespace as Markdown files in a zip written to `out`: free notes in folders that follow the
tree (a note's notes in a folder named like it), the pages of things under "Pages of things"."""
rows = db.rows(f"SELECT {FIELDS}, body FROM note_page WHERE space = $s", s=sid)
by_id = {p["id"]: p for p in rows}
titles = {i: p["title"] for i, p in by_id.items()}

def folder(p, seen=()):
if p.get("about"):
return ["Pages of things", THINGS.get(p["about"].split(":")[0], "Other")]
up = by_id.get(p.get("parent")) if p.get("parent") is not None else None
if not up or up["id"] in seen or len(seen) > DEPTH_MAX:
return []
return [*folder(up, (*seen, p["id"])), file_name(up["title"])]

used = set()
with zipfile.ZipFile(out, "w", zipfile.ZIP_DEFLATED) as z:
for p in sorted(rows, key=lambda r: (r.get("position") or 0, r["id"])):
base = "/".join([*folder(p), file_name(p["title"])])
path, n = f"{base}.md", 1
while path.casefold() in used: # two pages with one title in one place
n += 1
path = f"{base} ({n}).md"
used.add(path.casefold())
z.writestr(path, markdown(p, titles))
return len(rows)


SNIPPET = 160


Expand Down
36 changes: 36 additions & 0 deletions fastapi_backend/tests/api/test_notebook.py
Original file line number Diff line number Diff line change
Expand Up @@ -591,3 +591,39 @@ def test_daily_notes(client, env, db):
assert client.delete(f"/api/v1/notes/{root['id']}", headers=he).status_code == 200
again = client.post("/api/v1/notes/today", headers=he, json={"ns": "pods", "date": "2020-02-29"}).json()
assert again["parent"] not in (None, root["id"])


def test_markdown_export(client, env, db):
import io
import zipfile

he, hv, hx = env["he"], env["hv"], env["hx"]
plans = _new(client, he, title="Plans: Q4", summary="What's next", place="project")
old = _new(client, he, title="Hiring", parent=plans["id"])
_new(client, he, title="Budget", parent=plans["id"])
_new(client, he, title="budget", parent=plans["id"]) # the same title twice in one place
ada = _entity(db, "Ada Lovelace", "PERSON")
body = f"Talk to @[Ada](entity:{ada['id']}) about @[Hiring](page:{old['id']})."
client.patch(f"/api/v1/notes/{plans['id']}", headers=he, json={"body": body})
client.patch(f"/api/v1/notes/{old['id']}", headers=he, json={"title": "Hiring plan"})
_new(client, he, title="Ada's page", about=f"entity:{ada['id']}")

r = client.get(f"/api/v1/notes/{plans['id']}/markdown", headers=hv)
assert r.status_code == 200 and r.headers["content-type"].startswith("text/markdown")
assert "Plans Q4.md" not in r.headers["content-disposition"] and "Plans Q4.md" in r.headers["content-disposition"]
assert r.text.startswith('---\ntitle: "Plans: Q4"\n')
assert 'place: "project"' in r.text and 'summary: "What\'s next"' in r.text
assert r.text.endswith("Talk to Ada about [[Hiring plan]].\n") # the page's title now, the entity's name
assert client.get(f"/api/v1/notes/{plans['id']}/markdown", headers=hx).status_code == 404

r = client.get("/api/v1/notes/export?ns=pods", headers=hv)
assert r.status_code == 200 and r.headers["content-type"] == "application/zip"
names = sorted(zipfile.ZipFile(io.BytesIO(r.content)).namelist())
assert names == [
"Pages of things/Entities/Ada's page.md",
"Plans Q4.md",
"Plans Q4/Budget.md",
"Plans Q4/Hiring plan.md",
"Plans Q4/budget (2).md",
]
assert client.get("/api/v1/notes/export?ns=pods", headers=hx).status_code == 404
14 changes: 13 additions & 1 deletion nextjs-frontend/__tests__/notes-find.test.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ import "@testing-library/jest-dom";
import { Notes } from "@/app/openapi-client";
import { CommandPalette } from "@/components/app-shell/command-palette";
import { NotesHome } from "@/components/notes/notes-home";
import { TooltipProvider } from "@/components/ui/tooltip";

jest.mock("@/app/openapi-client", () => ({
Notes: {
Expand All @@ -19,6 +20,10 @@ jest.mock("next-auth/react", () => ({ useSession: () => ({ data: { accessToken:
const push = jest.fn();
jest.mock("next/navigation", () => ({ usePathname: () => "/notes", useRouter: () => ({ push }) }));
jest.mock("@/components/ui/toast", () => ({ useToast: () => jest.fn() }));
const download = jest.fn<Promise<void>, [string, string | undefined, string]>(() => Promise.resolve());
jest.mock("@/components/library/actions", () => ({
downloadWithToken: (url: string, token: string | undefined, name: string) => download(url, token, name),
}));
jest.mock("@/lib/hooks/session", () => ({
useArchive: () => ({
admin: false,
Expand All @@ -33,7 +38,7 @@ const m = (f: unknown) => f as jest.Mock;
const show = (ui: React.ReactNode) =>
render(
<QueryClientProvider client={new QueryClient({ defaultOptions: { queries: { retry: false } } })}>
{ui}
<TooltipProvider>{ui}</TooltipProvider>
</QueryClientProvider>,
);

Expand Down Expand Up @@ -112,6 +117,13 @@ describe("the Notes page", () => {
expect(screen.getByLabelText("Pinned")).toBeInTheDocument();
});

it("downloads every note as Markdown from its menu", async () => {
show(<NotesHome />);
fireEvent.keyDown(await screen.findByRole("button", { name: "More on Notes" }), { key: "Enter" });
fireEvent.click(await screen.findByRole("menuitem", { name: "Download all as Markdown" }));
await waitFor(() => expect(download).toHaveBeenCalledWith("/api/v1/notes/export?ns=home", "t", "home notes.zip"));
});

it("says when no note has the words", async () => {
m(Notes.searchPages).mockImplementation(() => ok({ total: 0, pages: [] }));
show(<NotesHome />);
Expand Down
3 changes: 2 additions & 1 deletion nextjs-frontend/__tests__/notes-today.test.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ import "@testing-library/jest-dom";
import { Notes } from "@/app/openapi-client";
import { NotesHome } from "@/components/notes/notes-home";
import { TodayNote, localDay } from "@/components/notes/today";
import { TooltipProvider } from "@/components/ui/tooltip";

jest.mock("@/app/openapi-client", () => ({
Notes: {
Expand Down Expand Up @@ -35,7 +36,7 @@ const m = (f: unknown) => f as jest.Mock;
const show = (ui: React.ReactNode) =>
render(
<QueryClientProvider client={new QueryClient({ defaultOptions: { queries: { retry: false } } })}>
{ui}
<TooltipProvider>{ui}</TooltipProvider>
</QueryClientProvider>,
);

Expand Down
14 changes: 12 additions & 2 deletions nextjs-frontend/__tests__/notes-tree.test.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,10 @@ let pathname = "/notes";
jest.mock("next/navigation", () => ({ usePathname: () => pathname, useRouter: () => ({ push }) }));
const toast = jest.fn();
jest.mock("@/components/ui/toast", () => ({ useToast: () => toast }));
const download = jest.fn<Promise<void>, [string, string | undefined, string]>(() => Promise.resolve());
jest.mock("@/components/library/actions", () => ({
downloadWithToken: (url: string, token: string | undefined, name: string) => download(url, token, name),
}));
let editor = true;
jest.mock("@/lib/hooks/session", () => ({
useArchive: () => ({ namespaces: [{ name: "home" }], namespace: "home", can: () => editor }),
Expand Down Expand Up @@ -138,12 +142,12 @@ describe("the notes tree", () => {
expect(m(Notes.unpinPage).mock.calls[0][0].path).toEqual({ pid: 4 });
});

it("lets someone who can only read pin, and nothing else", async () => {
it("lets someone who can only read pin and download, and nothing else", async () => {
editor = false;
show();
fireEvent.keyDown(await screen.findByRole("button", { name: "More for Trips" }), { key: "Enter" });
const items = await screen.findAllByRole("menuitem");
expect(items.map((i) => i.textContent)).toEqual(["Pin"]);
expect(items.map((i) => i.textContent)).toEqual(["Pin", "Download as Markdown"]);
expect(screen.queryByRole("button", { name: "New note inside Trips" })).not.toBeInTheDocument();
});

Expand All @@ -157,4 +161,10 @@ describe("the notes tree", () => {
expect(toast.mock.calls[0][0].title).toBe("Couldn’t pin it");
expect(screen.queryByRole("group", { name: "Pinned notes" })).not.toBeInTheDocument();
});

it("downloads a note as Markdown", async () => {
show();
await choose("Trips", "Download as Markdown");
await waitFor(() => expect(download).toHaveBeenCalledWith("/api/v1/notes/4/markdown", "t", "Trips.md"));
});
});
9 changes: 9 additions & 0 deletions nextjs-frontend/app/openapi-client/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -976,6 +976,11 @@ export type {
ExportBatchResultsError,
ExportBatchResultsErrors,
ExportBatchResultsResponses,
ExportNotesData,
ExportNotesError,
ExportNotesErrors,
ExportNotesResponse,
ExportNotesResponses,
ExportRecordingData,
ExportRecordingError,
ExportRecordingErrors,
Expand Down Expand Up @@ -2313,6 +2318,10 @@ export type {
PageHistoryErrors,
PageHistoryResponse,
PageHistoryResponses,
PageMarkdownData,
PageMarkdownError,
PageMarkdownErrors,
PageMarkdownResponses,
PageVersionData,
PageVersionError,
PageVersionErrors,
Expand Down
Loading
Loading