Skip to content

Validation: executor for v3 models, single-op harness, trust inventory - #1

Closed
yizhou7 wants to merge 7 commits into
v3from
validation-v3
Closed

yizhou7 wants to merge 7 commits into
v3from
validation-v3

Conversation

@yizhou7

@yizhou7 yizhou7 commented Nov 14, 2026

Copy link
Copy Markdown
Collaborator

Differential-testing work for v3, for review with zhuominc. Draft: the library change (first section) is the part that needs a decision; everything else lives in validation/.

Library change (lean/TVA)

So that an executor can run a model, each pick and rmw carries a solver (Solve), which Step ignores:

  • Prog.pick P solve k, Prog.rmw … solve …: the solver proposes a value with its proof (Ans.ok, Ans.empty), asking an oracle (Hints, a closed Query family) for what it cannot compute.
  • Ans.assume b: a value with no proof, at sites whose predicate mentions the platform (e.P): tl.exp, /, tl.dot, fp8 conversion, dot_add, dot_scaled_add, float atomic_add. The executor reports every assumed value.
  • Prog.plat q k is a definition (env then pick of PlatQ.Rel), not a constructor: no new Step rule; Safe.plat / SafeA.plat are theorems.

Meaning unchanged, no claim file changed, no proof file changed; all 41 oracle proofs check (validation/check_oracles.py, axioms propext, Classical.choice, Quot.sound).

Validation (validation/)

  • PLAN.md: the plan (phases, scope, the compiler's hidden parameters, what to decide). TRUST.md: what validates each device fact and tl.* definition today.
  • Exec.lean: exec / execLaunch with exec_sound / execLaunch_sound: what the executor reports, Step can reach, provided the assumed values satisfy their predicates. PICKS.md: what the executor does at each choice site.
  • ops/: single-operation tests of tl.* definitions through the executor against one-operation Triton kernels, with mutants to measure the inputs. device.py (GPU) has not been run yet; --self mode (model as device) shows every mutant caught where it differs from the definition.
  • COVERAGE.md (generated by coverage.py): per tl.* operation (121), what the executor does and which cases cover it.

To decide

See PLAN.md, "To decide with zhuominc": the change to Prog; evidence that names the previous library; recording check_gpu.py runs; which GPU; stating the stack in the trusted base.

🤖 Generated with Claude Code

yizhou7 and others added 7 commits November 11, 2026 18:48
- validation/PLAN.md: the plan for testing TVA v3 against the device (phases 0-5), the
  change to Prog that running models needs (solvers on pick, a plat effect), and the
  decisions to make with zhuominc.
- validation/TRUST.md: phase 0. For every device fact, tl.* definition, launch-semantics
  assumption and claim, what validates it today; the gaps, and questions for zhuominc.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Prog.pick P k erased P and its witness type in compiled code, so no executor
could run a model. pick now takes a Solve P (code that proposes a value with
its proof, asking typed Hints for what it cannot compute), and rmw a solver for
its update; Step ignores both, so no model's meaning changes. The platform
operations (math, div, cvt into fp8, dot, dot_scaled) move to a new
constructor plat q k (PlatQ), whose relation is the old env-then-pick predicate.

Solvers for the guards, hints, integer division and shifts, conversions (all
but into fp8), exact float operations, reductions (oracle tree, checked),
padding and integer atomics; Solve.none at the mixed platform/fusion sites,
scans, sort, reshape_any, max_constancy and float atomic_add
(validation/PICKS.md). Safe and SafeA gain a plat rule; the platform rules keep
their statements.

No claim file changes. All 41 oracle proofs check; two need a plat case in an
induction over Safe (recompute_wu_bf16, streamk_matmul_f16).

Pending zhuominc's review. docs/index/ not regenerated (make_index.py needs
flock, not on macOS).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Phase 3 of validation/PLAN.md. exec runs one program on a concrete memory (a
hash table, CMem, read through toMem) with an oracle for a fixed environment;
execLaunch runs a launch's programs one after another in grid order.
exec_sound: whatever exec reports, Step can reach; execLaunch_sound: a
finished launch is a Reach trace to a Done configuration with the reported
memory, a reported ub a trace to a StuckUB one.

tests/vec_add_f32.lean runs the claim's model on a small instance and checks
x + y (run_test.py). check_oracles.py checks every claim and oracle proof
against the built library without the sandbox. PICKS.md: the status of each
choice site.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tructor

A pick whose predicate mentions the platform (e.P) cannot be answered with a
proof. Ans gains a third answer, assume b: a value with no proof, which
Solve.assume takes from the oracle through a Query that carries the operands
(Query.plat, Query.convert, Query.atomicAdd). An executor reports what it
assumed; Step ignores solvers, so no meaning changes.

Prog.plat q k becomes a definition, env fun e => pick (q.Rel e.P)
(Solve.assume (.plat q)) k: the old env-then-pick pattern. Step.plat,
Step.platUB and the plat constructors of Safe and SafeA are gone; Safe.plat and
SafeA.plat are theorems with the statements the rules use, so no rule proof
changes, and the two claim proofs that needed a plat case are back to v3's text.

Solvers that assume: dot_add, dot_scaled_add (the dot with c as accumulator),
float atomic_add, tl.convert into fp8, and every Prog.plat site.

All 41 oracle proofs check unchanged (validation/check_oracles.py).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…erage

exec and execLaunch return, with their result, the values taken without proof
(Assumed); exec_sound and execLaunch_sound are conditional on each satisfying
its site's predicate.

validation/ops: one tl.* operation at a time, through exec. cases.py writes edge
and random inputs as bit patterns, device.py runs one-operation Triton kernels
on a GPU (not yet run), Ops.lean runs the library's definitions and compares
lane by lane with the relation each allows; undefined behaviour is localised
per lane, reductions match against candidate trees. Mutants (wrong copies of
definitions) measure the inputs; with --self the model stands in for the
device: every mutant is caught where it differs from the definition.

coverage.py writes COVERAGE.md, one row per tl.* operation (121): what exec
does at it, its cases and mutants. PLAN.md: phase 1 is the single-op harness;
assumed answers; plat as a definition. PICKS.md updated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…icResult

Solvers are stored in `pick` and `rmw` nodes, so `=` on continuations also
compared solvers. A continuation that used an atomic's old value only in a
solver (`Solve.ok old _`) counted as using the result in
`UsesSharedAtomicResult`, and made `Step.defer`'s premise vacuous, so a
deferred `pick` could choose per old value.

Add `Prog.SameUpToSolvers` (with `refl`, `of_eq`, `bind_left`) and use it in
both places, and in the "result unused" premises of `Safe.rmw`, `SafeA.rmw`,
`SafeA.shared` and the `Acc` lemmas, which are now weaker requirements.
Claim proofs only pass the premise along and are unchanged.

docs/index/ not regenerated (make_index.py needs flock).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant