Skip to content

enforce test quality beyond pass/fail #2

Closed

Description

@dominicnunez

Problem

The verification pipeline checks two things: were test files created/modified, and do they pass. This is necessary but not sufficient — an agent can satisfy both conditions with trivial assertions like:

test("it works", () => {
  expect(true).toBe(true);
});

The test quality rules in the prompt help guide the agent, but enforcement is honor-system. There's no programmatic check that tests actually exercise the code that was written.

Ideas

  • Coverage delta: run coverage before and after, require that new/changed lines have coverage above a threshold
  • Assertion density: check that test files have a reasonable ratio of expect() calls to test cases
  • Mutation testing: run a lightweight mutation pass to verify tests actually catch regressions (probably too slow for the loop, but could work as a willie-style post-pass)
  • AST analysis: check that test imports actually reference the files that were changed

Context

The current gate catches "forgot to write tests" and "tests are broken." The gap is "wrote tests that don't actually test anything." Coverage delta is probably the most practical first step.

Activity

  1. dominicnunez commented on Feb 21, 2026

    @dominicnunez
    OwnerAuthor

    Closed as duplicate of #24 - Phase 4 (Audit Pipeline) will include a test quality agent that verifies test coverage, meaningful assertions, edge cases, and test isolation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions