Skip to content

ci(scorecard): measure whether Lens meets its goals with make scorecard - #487

Merged
adeelahmad merged 1 commit into
mainfrom
claude/project-thread-yw34ry
Oct 17, 2026
Merged

adeelahmad merged 1 commit into
mainfrom
claude/project-thread-yw34ry

Conversation

@adeelahmad

Copy link
Copy Markdown
Owner

Requested by Adeel · project thread

Description

Before: nothing recorded whether a turn of work made Lens better, so agents couldn't back gains with before-and-after numbers.

After: make scorecard prints one JSON line with the commit, tag and release, and each number with the command or GitHub API call behind it (cmd), or why it isn't measured (not_measured). make scorecard SCORECARD=file.jsonl prints what changed since the file's last line, appends the new line, and exits 1 when a guard trips: journeys passing, journeys, tests or coverage went down, a measured number stopped being measured, or a clicks baseline changed without a new survey.

How: ci/scorecard/scorecard.py (stdlib only) reads the clicks baselines, counts test functions per project and collects the backend's tests with pytest when the backend is installed, downloads the newest backend-coverage and frontend-coverage artifacts from main, and reads pull requests and workflow runs from the REST API for the red-check rate, ask to merge, CI minutes per merged pull request and failed runs on main (last 7 days), plus stars, forks and outside issues and pull requests. It's a new Nx project (scorecard, test target) so CI runs its tests.

First line, taken on main at v0.13.0-17-g28eaed56 (67 GitHub calls, 1m41s):

Number Value
Journeys passing, computer / phone 31 of 31 / 31 of 31
Clicks + keys (computer), taps + keys + swipes (phone) 82 + 222 / 113 + 219 + 2
Test functions defined 2332 (backend 1296, frontend 968, clipper 48, installer 20)
Backend tests collected 1403
Coverage, backend / frontend 89.9% / 73.84%
Red-check rate 17.2% (36 of 209 pull requests)
Ask to merge (median) 0.39 h (196 merged)
CI minutes per merged pull request (median, wall time) 43.33
Failed runs on main 7 of 891 (CI 2, Clicks 4, pre-commit 1)
Stars, forks, outside issues / pull requests 0, 0, 0 / 0
Install to first answer, idle RAM, disk on arm64 not measured (follow-up in this issue)

Motivation and Context

Part of #481 (the Lens loop's scorecard). The timed install on an arm64 runner (install to first answer, idle RAM and disk) comes in a follow-up pull request; until then those report "not measured".

Types of changes

  • New feature (non-breaking change which adds functionality)

Checklist:

  • The title is a Conventional Commit.
  • My code follows the code style of this project (ci/local-ci.sh --jobs pre-commit,title passed).
  • I added or updated tests (21 in ci/scorecard/test_scorecard.py).
  • My change requires documentation updates, and I have updated docs/ accordingly (Contributing, Scorecard; AGENTS.md points at make scorecard).
  • Users need more than the title: n/a, a contributor tool.

Done questions (AGENTS.md)

  • Goal: merged on green CI next; the arm64 timing follows.
  • Agents: a command agents run; JSON out.
  • Zero setup: Python stdlib only; GitHub calls use GITHUB_TOKEN when set.
  • Fewest clicks and keystrokes: one command.
  • n/a Configurable: no app setting.
  • Light: nothing added to the install.
  • Private and secure: reads public repository data only; the token isn't passed on to artifact storage redirects. No new app surface.
  • n/a Cost: no model calls.
  • User decides: defaults unchanged.
  • n/a Upgrades: no stored data.
  • Honest: every number has its command or says it isn't measured; CI minutes are labelled wall time.
  • Tests, docs and generated files updated.
  • Open standards; no new dependencies.
  • Result reported with evidence; refine later: install timing on arm64 (follow-up), runner minutes from the jobs API if wall time proves too coarse.
  • Reuse: reads the existing clicks baselines and CI coverage artifacts instead of measuring again.

Generated by Claude Code

ci/scorecard/scorecard.py prints one JSON line per run: the commit, tag and
release, and each number with the command or API call behind it, or why it
isn't measured. Friction from the clicks baselines, tests defined and
collected, coverage from main's CI artifacts, red-check rate, ask to merge,
CI minutes per merged pull request, failed runs on main, stars, forks and
outside issues and pull requests. --log FILE compares with the file's last
line and appends; a guard trips (exit 1) when journeys, tests or coverage go
down, a measured number stops being measured, or a clicks baseline changes
without a new survey.

Part of #481.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SDzZtD95f3XeGHNfinM1qv
@adeelahmad adeelahmad self-assigned this Oct 17, 2026
@adeelahmad
adeelahmad merged commit 9a80e28 into main Oct 17, 2026
9 checks passed
@adeelahmad
adeelahmad deleted the claude/project-thread-yw34ry branch October 17, 2026 02:36
adeelahmad pushed a commit that referenced this pull request Oct 17, 2026
#469 merged while this was open: its line moves from In progress to the built list, with its checks. The
scorecard (#487) is merged too, so Planned keeps only what it doesn't measure yet.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MTK8aqzLBhdhfCmVf2LaWZ
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants