oraclebreadth case 1 (T=3) was a stale fixture, not a kernel defect

Lane: oracle_breadth case 1 (T=3) red at 1.851e-1 against BAR 1e-3 — decide by experiment whether 0a6998a (the production GVA/conv fix) is wrong for short T, or the fixture is stale. Branch / worktree: wt/breadth-t3 (off e7fd10a). Date: 2026-09-30. Verdict: the FIXTURE was stale. Production is correct. 0a6998a is exonerated and was never involved.

---

1. The question, and the answer in one line

tests/ref_f64_broad.bin had stopped being the output of the generator that ships beside it. ca45600 fixed a head-major layout bug in tools/gen_reference_f64.py, regenerated two of the three f64 fixtures, and left the 1000-case sweep behind. The “18.5% relative deviation at T=3” was the fixture’s error, not the kernel’s.

2. 0a6998a could not have caused it — readable in the source, no run needed

Both halves of that commit are excluded by arithmetic, before any experiment:

0a6998a half	why it cannot be the cause here
GVA head repeat (module.rs)	the whole block is inside if hv > h. This fixture is H=4, HV=4 (gen_reference_f64.py:154), so hv > h is false and the repeat never executes. The generator says so itself at :336 (if HV > H:) and :599 (“an identity in this crate’s own config (H == HV)”).
short-conv pad replicate → zeros (short_conv.rs)	this moved production toward the fixture, not away: the generator has zero-padded since 8162672 (gen_reference_f64.py:42-56, short_conv(..., pad="zeros") is the default). Reverting it re-introduces replicate against a zero-padded oracle.

So the brief’s experiment 1 (revert-scratch) was not needed, and the “T < kernel width” reading of a T=3 failure was a coincidence: T=3 is simply the first case in the sweep. The failing region is every T > 1.

3. The decisive measurement: the fixture does not regenerate

The header told me to check regeneration and then reach for git diff --exit-code. Doing that, in crates/burn-gdn2:

$ python3 tools/gen_reference_f64.py --broad --out /tmp/regen.bin
$ cmp tests/ref_f64_broad.bin /tmp/regen.bin
differ: byte 155610

Parsing both files (a 40-line comparator, max|A-B| per tensor / per case):

header A d=64 h=4 hk=16 hv=4 expand=1.5     header B d=64 h=4 hk=16 hv=4 expand=1.5
-- weights (max |A-B|) --            (none: all 17 tensors bit-identical)
-- inputs:  max |A-B| over all 1000 cases = 0.000000e+00
-- outputs: 1000/1000 cases differ
   case    0 T=  1  max|A-B|=7.459311e-17  scale=1.067e-01  rel=6.994e-16
   case    1 T=  3  max|A-B|=2.223209e-02  scale=1.201e-01  rel=1.851e-01   <-- THE FAILING CASE
   case    2 T=  6  max|A-B|=4.879912e-02  scale=2.857e-01  rel=1.708e-01

The weights and the inputs are bit-identical; only the outputs move. And at case 1 the fresh generator differs from the committed bytes by 2.223209e-02 absolute / 1.851e-01 relative / scale 1.201e-01 — the failing test’s number to four significant figures, including the output scale. Same input, same weights, the deviation lives entirely in the expected values.

3a. Which generator wrote it — proved, not inferred

Revert one line of the generator to its pre-ca45600 form and re-diff:

# current (ca45600, correct)
g = g.T.reshape(t, H, HK).transpose(1, 0, 2)
# pre-ca45600
g = g.T.reshape(H, t, HK)

$ python3 /tmp/gen_old.py --broad --out /tmp/oldgen.bin     # the one line reverted
$ (compare against the committed fixture)
   case 0 T=1  rel=6.994e-16     case 1 T=3  rel=6.355e-16
   case 2 T=6  rel=8.743e-16     case 3 T=11 rel=8.009e-16    ... all 1000 cases

All 1000 cases agree to 4.1e-16 .. 9.7e-16 RELATIVE — f64 round-off. The committed ref_f64_broad.bin is the pre-ca45600 generator’s output, to the last bit it can be pinned to.

ca45600’s diff on the generator is one line of maths plus the comment block that describes the very bug it fixes:

g = g.T.reshape(H, t, HK)
g = g.T.reshape(t, H, HK).transpose(1, 0, 2)

and its file list is the whole story:

 tools/gen_reference_f64.py  | 104 ++++++-
 tests/ref_f64.bin           | Bin 467997 -> 467997 bytes     <- regenerated
 tests/ref_f64_faults.bin    | Bin 251024 -> 286887 bytes     <- regenerated
 tests/ref_f64_broad.bin     | (absent)                      <- NOT regenerated

A head-major reshape of a token-major buffer: identical at t == 1, wrong at every t > 1. The same defect class as ff7cd57, which the same comment block names.

3b. Only one of three fixtures is stale

fixture	vs the current generator	verdict
tests/ref_f64.bin (18-case)	≤ 9.7e-16 rel	current
tests/ref_f64_faults.bin (8 faults)	≤ 4.4e-15 rel, input fnv1a64 matches	current
tests/ref_f64_broad.bin (1000-case)	up to 8.94e-01 rel	STALE

4. Why a day went into the wrong place: the old numbers were the fixture’s

The previous lane’s oracle_breadth.rs STATUS block reported these as kernel measurements. All four are reproduced by comparing the current generator against the stale committed bytes — to five or six figures:

reported as a kernel reading	actually the stale-fixture signature
976/1000 cases over the bar	976/1000 — exact
worst 8.9402e-01 rel at T=3	8.940234e-01 at case 715 (T=3) — exact
first failing case: 1 (T=3), 1.851e-01 rel	1.851e-01 at case 1 — exact
“exactly 24 of the 1000 cases pass and they are exactly the T=1 ones (i % 42 == 0)”	exactly the 24 with i % 42 == 0, all T=1 — exact
per-case output scale 3.93e-02 / 3.21e-01 / 5.35e-01 (min/median/max)	3.930e-02 / 3.290e-01 / 5.487e-01 — min matches, median and max do not
nearest wrong formula 3.909822e-01 (no-erase-gate at T=8), 391× the bar	5.266142e-01 (no-erase-gate at T=35); the true nearest is now 4.084390e-01, output-gate-sigmoid at T=2, 408× the bar

Two of those are the smallest margins in the file — the two numbers that carry the “the bar bites a wrong formula” claim. Both moved. The five large margins (conv-padding 4.94, transposed-proj 3.30, no-scale 2.69, read-before-write 1.51, no-write-gate 1.02) are unchanged, because a fault that moves the answer by O(1) is insensitive to a 1e-1 error already in the expected value. The numbers that moved are exactly the ones that were small, and the ones that were small are exactly the ones the bar’s credibility rests on.

The T=1 coincidence is the fingerprint of the defect class: head-major and token-major splits index identically at t == 1, so precisely the degenerate cases agreed. That is what made a layout bug present as a state-carry bug, and why “the fault is in the carry, project is clean” was consistent with the available evidence and wrong anyway. The hand-dumped T=2 carry agreeing to 4.6e-07 was never in tension with anything: the carry was always right; the fixture it was being compared against was not.

The one number the previous block flagged as untrustworthy

“A separate probe that called the crate’s own fused_recurrent_forward reported a final state 1.34e-01 off on the same input, which contradicts that and which I could NOT REPRODUCE. I am not claiming either number.”

The number it could not reproduce is 1.34e-01. The stale fixture’s error at the worst case is 8.94e-01 and at case 1 is 1.85e-01. A probe reading a stale fixture is 1.34e-01 from the truth for the same reason everything else was.

4a. An independent prediction of GREEN, made before the test ran

The diagnosis says the Rust is on the fresh generator’s side of the disagreement. That is a claim about code I have not executed, so it was worth a second instrument that does not share a build with the test: transcribe src/module.rs::project and src/kernel/fused_recurrent.rs op-for-op into numpy in float32, with the fixture’s f32-rounded weights (which is what the loader hands the model, since the fixture stores f32) and the f32 input, and compare it to the f64 oracle.

  case    0 T=  1  f32 simulation vs the f64 oracle: 2.313e-07 rel
  case    1 T=  3  f32 simulation vs the f64 oracle: 3.659e-07 rel   <- the failing case
  case    7 T=  2  f32 simulation vs the f64 oracle: 4.223e-07 rel
  case  715 T=  3  f32 simulation vs the f64 oracle: 6.103e-07 rel   <- the worst stale case
  case  999 T= 13  f32 simulation vs the f64 oracle: 4.517e-07 rel

Every case lands at 2.3e-07 .. 6.1e-07 relative, ~1600x inside the 1e-3 bar — and that band is exactly the f32-noise band the test header already measured independently for the per-head stages (1.3-5.2e-07) and the short conv (1.4e-07 .. 2.9e-07). Two unrelated arithmetic routes to the same number is a prediction, and it is falsifiable: had production actually been wrong for T < kernel width, this would have read O(1) at case 1 and the fix would have been to change src/.

AND THE REPRODUCER CONVINCED ME OF THE OPPOSITE FIRST, WHICH IS THE POINT. My first version of that script read 9.374e-01 at case 1 and 1.212e+00 at case 7 — indistinguishable from a kernel defect, and with the same signature as the one under investigation: right at T=1, O(1) wrong from T=2 on. The bug was mine: I reshaped the [HV, T, VH] readout straight to [T, VD] without the transpose to [T, HV, VH] first, which reinterprets flat memory across heads and tokens. That is the same head-major/token-major class as ca45600’s generator bug, it is silent on T=1 by construction, and I walked into it while writing the instrument for it. The class is cheap to walk into and worth a named gate rather than a code review; the checker added in §6 is that gate, and this paragraph is the receipt for why it is needed.

5. Production is right

src/module.rs’s to_4d is reshape([b, tt, n, d]).permute([0, 2, 1, 3]) — token-major split, then permute, i.e. the fixed form. src/kernel/fused_recurrent.rs transcribes Eq. 9/10 with the readout taken from the state after the write (state = state + k_tᵀ·v_new then out = (state · q_tᵀ).sum_dim(2)), which is what gen_reference_f64.py:367 (src = Sb if fault == "read-before-write" else S) demands. The pad is zero (short_conv.rs:58, Tensor::zeros). The two fixes 0a6998a made are both present and both correct.

6. The fix

1. Regenerate tests/ref_f64_broad.bin from the current generator. Licensed by §3/§3a: the disagreement is explained and the direction is proved, which is the bar the fixture’s own previous header set (“regenerating would have re-committed the bug as the truth”).
2. tools/check_f64_fixtures.py (new) — the gate that was missing. No cargo test can see a fixture disagreeing with its generator, because the committed bytes are what the test compares against. It regenerates all three f64 fixtures into a temp dir and compares: weights and inputs bit-identical (deterministic stream; any difference is a changed input), outputs to OUT_TOL = 1e-12.
3. Hooked into tools/lib_gate.sh, before the cargo cells, so it runs on every wt.sh test and on CI’s lib_gate.sh. Needs only numpy, no build.
4. Corrected both test headers. The oracle_breadth.rs STATUS block now carries the real finding and the exoneration; the 3.909822e-01, 3.21e-01, 5.35e-01, 8.9402e-01 and “seven wrong formulas” numbers are replaced with re-measured ones; and the git diff --exit-code regeneration recipe is gone from both files.

Why OUT_TOL = 1e-12 and not a byte-diff

The old recipe was wrong twice. The f64 outputs are numpy reductions whose summation order depends on the array’s memory layout, so a correct regeneration agrees to ~7e-16 rather than byte-for-byte — measured: ref_f64.bin ≤ 9.7e-16, ref_f64_faults.bin ≤ 4.4e-15. git diff --exit-code fails on a correct regeneration, so the person following it learns to ignore it, which is one of the two ways this went unnoticed for a day. The bar 1e-12 sits 3.6× above the worst observed round-off and 4.1e+11 below the smallest wrong formula (4.08e-01): margin on both sides, so it cannot pass a semantic change and cannot trip on rounding. A byte-diff is not a measurement of anything.

7. red → green → red → green

See §8 for the measured run. The two directions:

• red: the committed ref_f64_broad.bin (pre-ca45600) — check_f64_fixtures.py reports STALE 8.940234e-01 at case 715, and oracle_breadth fails at case 1 (T=3) 1.851e-01 ≥ BAR 1e-3.
• green: the regenerated fixture — the checker passes, and both arms are inside the bar.
• The demonstration is re-runnable in one line each way, and both are printed in the test header.

8. Runs — all four, this worktree, wt/breadth-t3 at e7fd10a

Every run through tools/build_lock.sh run <label> -- <cmd> (§1.5, one heavy thing at a time), --release, CARGO_TARGET_DIR pointed at the shared burn-fused target dir because /home had 5.6 GB free and a cold worktree build needs ~40 GB. The 1000-case sweep takes 7-14 s in release, not the ~160 s the brief expected; the queue for the lock, not the test, was the cost.

RED 0 — the committed (pre-ca45600) fixture

$ python3 tools/check_f64_fixtures.py
  STALE FIXTURE. The generator and the committed bytes disagree:
    tests/ref_f64_broad.bin
    OUTPUTS DIFFER: worst 8.940234e-01 rel at case 715 (T=3), bar 1e-12

$ cargo test --release -p burn-gdn2 --features binary-tests --test oracle_breadth
thread 'gdn2_1000_cases_match_the_f64_oracle' panicked at tests/oracle_breadth.rs:284:9:
case 1 (T=3): max relative deviation 1.851e-1 >= BAR 1e-3 (output scale 1.201e-1, i.e. 2.223e-2 absolute)
test result: FAILED. 2 passed; 1 failed; finished in 6.97s

The lane’s failure, reproduced exactly, from the same bytes.

GREEN — regenerate from the current generator

$ python3 tools/gen_reference_f64.py --broad
wrote tests/ref_f64_broad.bin: 1000 cases, d=64 h=4 hk=16 hv=4 expand_v=1.5, T in 1..38
$ python3 tools/check_f64_fixtures.py
OK  every committed ref_f64*.bin is the output of this generator (f32 exact, f64 to 1e-12).

$ ... --test oracle_breadth -- --nocapture
fixture ref_f64_broad.bin: d=64 h=4 hk=16 hv=4 expand_v=1.5 -> value head 24, VD=96, 1000 cases, 17 tensors
FusedRecurrent vs the f64 oracle, 1000 cases: worst 8.512e-07 rel at T=3 (scale 1.110e-1, 9.451e-08 abs) against BAR 1e-3
test result: ok. 3 passed; 0 failed; finished in 7.86s

$ ... --test oracle_chunk -- --nocapture
Chunk, all 5 chunk sizes x 1000 cases: worst 8.512e-07 rel (BAR 1e-3)
test result: ok. 2 passed; 0 failed; finished in 6.41s

$ cargo test --release -p burn-gdn2 --test ref_f64
test result: ok. 2 passed; 0 failed; finished in 0.12s

RED 2 / GREEN 2 — the round trip, same session, no other change

direction	oracle_breadth	checker
stale bytes installed	FAILED, case 1 (T=3) 1.851e-01	STALE 8.940234e-01 at case 715
regenerated	ok 3/3, worst 8.512e-07	OK, f32 exact, f64 to 1e-12

Magnitudes: 1.851e-01 → 8.512e-07, a factor of 2.2e+05. The residue is f32 rounding, and the prediction of §4a (an independent numpy-f32 transcription of the same op order) was 3.7e-07 .. 6.1e-07 on these five cases — the same band, agreeing to within 1.4x, from a completely different arithmetic.

oracle_chunk shares the root: yes, exactly

Same file, same load_broad(), same weights, same BAR. It read worst 8.512e-07 on the regenerated fixture at all five chunk sizes (4/8/16/32/64), which is also the first time that arm has been able to say anything about its own arithmetic — under the stale fixture every size was swamped and the previous header said so (“the honest state of this arm today is blocked”). The chunked arm and the fused arm agree to the printed digit, which is a result about the chunked WY factorisation and not about a fixture.

ref_f64.rs (the 18-case test) is green too, and that retires the “pre-existing” claim

2 passed; 0 failed. Its fixture is current (§3b), and §9’s arithmetic predicted this: the 9.227e-02 that its old header called a pre-existing state-carry defect is 9.227411e-02, the pre-ca45600 generator against the current one. So the third red in this cluster had the same single cause, and ca45600 had already fixed it by regenerating that fixture — the measurement was simply taken before that and read as a kernel defect.

The new gate can fail — three mutants, measured

python3 tools/check_f64_fixtures.py, each mutant applied and reverted in the worktree:

mutant	verdict
baseline	OK every committed ref_f64*.bin is the output of this generator
the ca45600 generator bug re-injected	STALE ... worst 5.577544e-01 rel at case 16 (T=65) (matrix) and 1.224407e+00 (decay-sign) (faults)
one flipped bit in a committed output	STALE ... worst 1.088110e-09 rel at case 5 (T=37) — 1000x above the 1e-12 bar
one flipped bit in a committed weight	STALE ... q_proj: WEIGHTS DIFFER (max 4.413e-02)
reverted	OK

A gate nobody has seen fail is the defect class this layer exists to kill, so this one has been made to fail in all three directions, and the single-bit cases are the interesting ones: they are what the git diff --exit-code recipe this replaces would have missed in one direction (it fires on the last bit of an output, which is 1e-16 of noise, i.e. it fires on nothing real) while telling you nothing about which side moved.

9. Follow-ups, not done here

• The “PRE-EXISTING, PROVEN BY REBUILDING THE BASE” claim in the old oracle_breadth.rs header was the same artifact, and it was load-bearing. It read:

  cargo test --release -p burn-gdn2 --test ref_f64 → FAILED, FusedRecurrent case 1 (T=2): max relative deviation 9.227e-02. That is dbfa4ca with none of this work in it. The f64 layer’s own test was already red before this commit.

  That is the pre-ca45600 generator against the current one, on the 18-case matrix:

  case  0 T=  1  rel 0.000000e+00     case  1 T=  2  rel 9.227411e-02   <- the reported red
  case  2 T=  3  rel 1.513291e-01     case 15 T= 64  rel 5.529454e-01

  9.227411e-02 against a reported 9.227e-02. ca45600 regenerated ref_f64.bin, so the red this header called pre-existing was a stale fixture that has since been regenerated — and “rebuild the base to prove it is pre-existing” cannot tell the two apart, because a stale committed fixture is pre-existing by construction. CONFIRMED, §8: cargo test --release -p burn-gdn2 --test ref_f64 is 2 passed; 0 failed on this tree. The whole “state-carry defect” narrative retires: all three reds in this cluster had the one cause, and the one that survives as a measurement (8.512e-07 against a 1e-3 bar) says the kernel was right all along.

• Nothing anywhere in the tree checks a fixture against its generator beyond the three f64 files I covered. ref_bwd_f64*.bin, ref_bwd_f64_carry.bin, ref_bwd_f64_faults.bin and every other crate’s oracle/*.bin have the same exposure and the same missing check. check_f64_fixtures.py is deliberately written so the second one is a copy with two names changed.

• gen_reference_f64.py writes tests/ref_f64_faults.bin as a side effect of the default --out run, and its default --faults-out is a committed path. Running the documented “regenerate” command therefore rewrites a committed fixture whether or not you meant to. That is how one of the three could be regenerated and another missed without anyone noticing the mechanism. The default should be None.