Repository navigation
ci(scorecard): measure whether Lens meets its goals with make scorecard - #487
Merged
Merged
Conversation
ci/scorecard/scorecard.py prints one JSON line per run: the commit, tag and release, and each number with the command or API call behind it, or why it isn't measured. Friction from the clicks baselines, tests defined and collected, coverage from main's CI artifacts, red-check rate, ask to merge, CI minutes per merged pull request, failed runs on main, stars, forks and outside issues and pull requests. --log FILE compares with the file's last line and appends; a guard trips (exit 1) when journeys, tests or coverage go down, a measured number stops being measured, or a clicks baseline changes without a new survey. Part of #481. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SDzZtD95f3XeGHNfinM1qv
16 of 17 tasks
adeelahmad
pushed a commit
that referenced
this pull request
Oct 17, 2026
#469 merged while this was open: its line moves from In progress to the built list, with its checks. The scorecard (#487) is merged too, so Planned keeps only what it doesn't measure yet. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MTK8aqzLBhdhfCmVf2LaWZ
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Requested by Adeel · project thread
Description
Before: nothing recorded whether a turn of work made Lens better, so agents couldn't back gains with before-and-after numbers.
After:
make scorecardprints one JSON line with the commit, tag and release, and each number with the command or GitHub API call behind it (cmd), or why it isn't measured (not_measured).make scorecard SCORECARD=file.jsonlprints what changed since the file's last line, appends the new line, and exits 1 when a guard trips: journeys passing, journeys, tests or coverage went down, a measured number stopped being measured, or a clicks baseline changed without a new survey.How:
ci/scorecard/scorecard.py(stdlib only) reads the clicks baselines, counts test functions per project and collects the backend's tests with pytest when the backend is installed, downloads the newestbackend-coverageandfrontend-coverageartifacts from main, and reads pull requests and workflow runs from the REST API for the red-check rate, ask to merge, CI minutes per merged pull request and failed runs on main (last 7 days), plus stars, forks and outside issues and pull requests. It's a new Nx project (scorecard,testtarget) so CI runs its tests.First line, taken on main at
v0.13.0-17-g28eaed56(67 GitHub calls, 1m41s):Motivation and Context
Part of #481 (the Lens loop's scorecard). The timed install on an arm64 runner (install to first answer, idle RAM and disk) comes in a follow-up pull request; until then those report "not measured".
Types of changes
Checklist:
ci/local-ci.sh --jobs pre-commit,titlepassed).ci/scorecard/test_scorecard.py).docs/accordingly (Contributing, Scorecard; AGENTS.md points atmake scorecard).Done questions (AGENTS.md)
GITHUB_TOKENwhen set.Generated by Claude Code