Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,12 @@ The backend (`fastapi_backend`) and the frontend (`nextjs-frontend`) are version

## Unreleased

- **The Ask button no longer covers buttons.** The floating Ask button at the bottom right used to sit on whatever was
there, such as Import on the Import page. Now, when a button or field is under it and scrolling can't move it clear
(a row of buttons pinned to the bottom of the screen, or the last button on a page scrolled to its end), the Ask
button rises above it, and steps out of sight when there's no room to rise. Dialogs, drawers and toasts drawn over
it don't move it. The clicks survey now fails a journey whose control the Ask button covers, and checks every page
for it ([Chat on any page](docs/assistant.md)).
- **Google Calendar as a sensor.** Add a Google Calendar connection on the Sources page (or `POST /api/v1/sources`
with type `gcal`): each calendar is a folder and each event becomes text, as an iCal feed's do, read-only through
Google's Calendar API. It signs in as a Google account (an OAuth client's authorized user JSON) or as a service
Expand Down
9 changes: 6 additions & 3 deletions ci/clicks/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,10 +32,13 @@ rclone must be on the PATH, as it is in the Lens image (the Clicks workflow copi
navigation).
- **Outside the browser**: steps in the terminal, the assistant being connected or the browser's print dialog are
counted on the page `(outside the browser)`.
- A step whose control is covered by something else (the floating Ask button) is still one click, marked
"covered" in the steps.
- A step whose control is covered by something else (a toast) is still one click, marked "covered" in the steps. A
control the floating Ask button covers fails the journey: the button rises above controls in its corner, or steps
aside, so that's a bug.
- **Per page**: reach is clicks from Home through the sidebar (sections and tabs are two); controls are the buttons,
links and fields in the page's main area on a fresh install, not counting navigation.
links and fields in the page's main area on a fresh install, not counting navigation. Each page is also scrolled to
its end and checked for controls the floating Ask button lies over; any fails the run ("Covered by the Ask button"
in the results).

Journeys run in order on the same Lens, each from Home unless it continues the one before (setup does). A journey
that can't be done at all (on a phone, say, where a control is hidden) fails with the reason; it is in the baseline
Expand Down
4 changes: 4 additions & 0 deletions ci/clicks/journeys.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -670,6 +670,10 @@ export const JOURNEYS = [
from: "/",
async run(ctx) {
const { page, meter } = ctx;
// An entity is in the graph once something links it twice (graph.min_edge_weight), which takes import-file's
// pi-episode as well; a slow machine can still be reading it here, so wait for the jobs to finish (not counted)
const idle = page.getByRole("button", { name: /^Activity: (Nothing running|\d+ failed)$/ });
await madeInBackground(ctx, "/", idle, "/", "the imports' jobs finished");
await go(ctx, "Graph");
const find = page.getByRole("combobox", { name: "Find in graph" });
await page.getByRole("region", { name: "Graph" }).waitFor();
Expand Down
13 changes: 11 additions & 2 deletions ci/clicks/meter.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,13 @@ function hasFocus(el) {
return Boolean(el.isContentEditable && active?.contains(el) && caret && el.contains(caret));
}

/** The floating Ask button lies over the middle of `el`, where a click lands. */
function coveredByAsk(el) {
const r = el.getBoundingClientRect();
const top = document.elementFromPoint(r.left + r.width / 2, r.top + r.height / 2);
return Boolean(top?.closest("[data-ask-button]"));
}

export class Meter {
/** `touch`: a phone, where clicks are taps and bringing a control into view takes a swipe. */
constructor(page, { touch = false } = {}) {
Expand Down Expand Up @@ -103,9 +110,11 @@ export class Meter {
try {
await this.tapOrClick(locator, { trial: true, timeout: 2000 });
} catch {
// Something still lies over it (the floating Ask button, a toast). The step notes it; the click still counts
// one, done here with the keyboard so it reaches the control rather than what covers it.
// The floating Ask button rises above controls (or steps aside), so it covering one is a failure. Anything else
// lying over it (a toast) is noted in the step; the click still counts one, done here with the keyboard so it
// reaches the control rather than what covers it.
await locator.waitFor({ state: "visible" });
if (await locator.evaluate(coveredByAsk)) throw new Error(`${what} is covered by the floating Ask button`);
const step = this.journey?.steps.at(-1);
if (step) step.what += " (covered by something else on the page)";
await locator.focus();
Expand Down
5 changes: 5 additions & 0 deletions ci/clicks/report.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -100,6 +100,11 @@ export function markdown(result, base) {
lines.push(
`| **All** | **${result.journeys.length} features** | **${sum.clicks}** | **${sum.keys}** | **${sum.prompts}** |${sw(`**${sum.swipes}**`)} **${sum.clicks + sum.keys + sum.prompts + sum.swipes}** | |`,
);
const covered = (result.pages || []).filter((p) => p.askCovers?.length);
if (covered.length) {
lines.push("", "## Covered by the Ask button", "", "With the page at its end, the floating Ask button lies over these:", "");
for (const p of covered) lines.push(`- ${p.label} (\`${p.page}\`): ${p.askCovers.join(", ")}`);
}
lines.push("", "## Per module", "", `| Module | Features | ${Clicks} | Keys | Prompts |${sw("Swipes")}`, `|---|--:|--:|--:|--:|${sw("--:")}`);
for (const m of modules) {
const js = result.journeys.filter((x) => x.module === m);
Expand Down
41 changes: 38 additions & 3 deletions ci/clicks/survey.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,41 @@ async function surveyPages(page, base, phone) {
return seen.filter((el) => el.offsetParent !== null && !el.closest("[hidden], nav, details:not([open]) > :not(summary)")).length;
});
const settle = () => page.waitForTimeout(1200);
// The controls the floating Ask button covers with the page (and anything scrolling in it) at its end, where
// scrolling can't carry them clear: it should rise above them or step aside, so any is a failure.
const askCovers = async () => {
await page.evaluate(() => {
window.scrollTo(0, document.documentElement.scrollHeight);
for (const el of document.querySelectorAll("main *"))
if (el.scrollHeight > el.clientHeight && /(auto|scroll)/.test(getComputedStyle(el).overflowY)) el.scrollTop = el.scrollHeight;
});
await page.waitForTimeout(600); // the button moves on the next frame, then slides
return page.evaluate(() => {
const ask = document.querySelector("[data-ask-button]");
if (!ask || getComputedStyle(ask).visibility === "hidden") return [];
const a = ask.getBoundingClientRect();
const out = [];
const all = "a[href], button, input:not([type=hidden]), select, textarea, [role=tab], [role=switch], [role=checkbox], [role=radio]";
for (const el of document.querySelectorAll(all)) {
if (el.closest("[data-ask-button], [data-page-chat]")) continue;
const r = el.getBoundingClientRect();
if (r.height > innerHeight / 3) continue; // a big field: the button covers a corner of it (ask-place.ts)
const [x0, x1, y0, y1] = [Math.max(r.left, a.left), Math.min(r.right, a.right), Math.max(r.top, a.top), Math.min(r.bottom, a.bottom)];
if (x1 - x0 < 4 || y1 - y0 < 4) continue;
// the button is on top there, and right under it is this control (not one a closed fold keeps out of sight)
const [top, ...under] = document.elementsFromPoint((x0 + x1) / 2, (y0 + y1) / 2);
const next = under.find((e) => !e.closest("[data-ask-button]"));
if (top?.closest("[data-ask-button]") && next && el.contains(next))
out.push((el.getAttribute("aria-label") || el.textContent || el.tagName).replace(/\s+/g, " ").trim().slice(0, 60));
}
return out;
});
};
const visit = async (entry) => {
entry.askCovers = await askCovers();
if (entry.askCovers.length) console.log(`FAIL ${entry.page}: the Ask button covers ${entry.askCovers.join(", ")}`);
pages.push(entry);
};
await page.goto(base + "/", { waitUntil: "domcontentloaded" });
await settle();
if (phone) await page.getByRole("button", { name: "Open navigation" }).tap();
Expand All @@ -108,7 +143,7 @@ async function surveyPages(page, base, phone) {
await page.waitForLoadState("domcontentloaded");
await settle();
const key = pageKey(page.url());
pages.push({ page: key, label: item.label, reach: 1 + menu, controls: await controls() });
await visit({ page: key, label: item.label, reach: 1 + menu, controls: await controls() });
// a second level: settings sections, and tabs that are links (Admin, Sensors, Pipelines)
const sub = await page
.locator("main")
Expand All @@ -120,7 +155,7 @@ async function surveyPages(page, base, phone) {
if (!s.href || pages.some((p) => p.page === pageKey(base + s.href))) continue;
await page.goto(base + s.href, { waitUntil: "domcontentloaded" });
await settle();
pages.push({ page: pageKey(page.url()), label: `${item.label} › ${s.label}`, reach: 2 + menu, controls: await controls() });
await visit({ page: pageKey(page.url()), label: `${item.label} › ${s.label}`, reach: 2 + menu, controls: await controls() });
}
}
return pages;
Expand Down Expand Up @@ -214,7 +249,7 @@ async function main() {
for (const d of worse) console.log(`worse: ${d.id} ${d.what}`);
if (worse.length && a.failOnRegression) process.exitCode = 1;
}
if (failed) process.exitCode = 1;
if (failed || pages.some((p) => p.askCovers.length)) process.exitCode = 1;
}

await main();
4 changes: 3 additions & 1 deletion docs/assistant.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,9 @@ then.
question (up to 12,000 characters of page text and 4,000 of the highlight) as what the person is looking at, not as
citable excerpts. The question keeps the page's address, title and highlight, shown above it; the page's text isn't
stored. A recording's transcript and document keep their own selection toolbar, whose Ask in chat goes to the
recording's chat.
recording's chat. The Ask button stays clear of the page's own buttons: when one is under it and scrolling can't
move it away (a row of buttons pinned to the bottom of the screen, or the last button with the page at its end), it
rises above them, or steps out of sight when there's no room to rise.
- **No model configured:** chat returns the best-matching passages instead. Anyone signed in can see whether a model
is set up, and which (`GET /chats/capabilities`), so the app says so before the first question.

Expand Down
8 changes: 7 additions & 1 deletion docs/frontend.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,13 @@ four-colour loop (Transcribe blue, Diarize red, Analyze green, Summarize and Rep
namespace switcher sits at the top of the menu, Import is an icon in the top bar, settings sections open as a list
from the current section's name, tabs wrap, and wide tables (sensors, connections) become cards. A form's main buttons
take `phone-actions` (`app/globals.css`) to stay pinned at the bottom of the screen, clear of the floating Ask button;
a Dialog's actions stay in sight on their own. The [clicks
a Dialog's actions stay in sight on their own.
* **The floating Ask button** (`components/page-chat/ask-place.ts`) finds what it would cover with the browser's own
hit testing, through a transparent probe just under it, and rises above any control there that scrolling can't
carry clear (one in a bar that's fixed or stuck to the bottom, or with the page at its end), or steps out of sight
when that would take it past the middle of the screen, so a new page needn't leave room for it (`phone-actions` and
the Settings save row still do, and it stays put there). The clicks survey fails a journey whose control it covers,
and checks every page with the page scrolled to its end. The [clicks
survey](https://github.com/adeelahmad/lense/blob/main/ci/clicks/README.md) measures every journey on a phone too.
* **The role rule:** actions a person can't take stay visible, disabled, with a tooltip saying who can
(`needRole()`); namespaces they have no role in never appear. `useArchive()` gives `can(role, ns)` and the
Expand Down
Loading
Loading