2026-10-09. One row per defect, each with its measurement, its owner, and
the workaround’s measured effect. Evidence levels are named, because the
difference between “proven” and “correlated” is what decides where the fix
goes — and half of today’s work was refuting my own leading hypotheses.
| id |
defect |
owner |
evidence |
level |
| A1 |
The autotuner selects a numerically different candidate per process for the same shape. 50 of 125 keys differ between two runs of one program; two keys separate our outcomes 4 exploding / 2 clean, winners’ medians 1.6 % apart (matmul_simple_cyclic_mma_multi_rows vs matmul_ordered_double_mma_partition_k2_row_count8_rows_per_plane2) |
tracel-ai/cubek (candidates) + tracel-ai/cubecl (selector) |
decisions sink + the recipe in the drafted issue |
proven that the choice varies; correlated 4/2 that it is the corruption |
| A2 |
The persistent autotune cache never materialises: target/environment is never created (3 runs from one cwd, 125 fresh decisions each, 0 files) although 2 692 persistence symbols are linked |
tracel-ai/cubecl |
ls target/environment after runs |
proven (the feature does not run) |
| A3 |
A narrow-output matmul reads never-written global memory: 100 initcheck events at out_features = 2, 0 at 64; the value is masked out of the result (0.000e0 against a materialised rhs at 2/3/4) |
tracel-ai/cubek |
matmul_uninit_probe::narrow_* under --tool initcheck |
proven (their own tool); harmless in value |
| A4 |
bf16×bf16 matmul is no longer refused on 0.22: it runs and returns a BF16 tensor matching fp32 to 3.09e-3 relative (bf16 precision). The pre.4 tripwire bf16_matmul_is_refused is red against it |
tracel-ai/burn |
bf16_matmul_on_0_22_is_a_silent_cast_or_garbage |
proven behavior change; upstream must say which it is (new path vs silent cast) |
| A5 |
ops_grad_cuda is red on 0.22: q_proj.weight autodiff −1.174e-1 vs central differences −1.021e-1, rel 1.5e-1 over a 5e-2 bar |
tracel-ai/burn (ops path) or our gate |
cargo test -p burn-kda --features cuda,autodiff --test ops_grad_cuda |
our instrument is the first suspect: fd(h) and fd(h/10) disagree by 2×, so the reference is at its noise floor. Fix the reference before accusing burn |
The drafted issue text for A1/A2/A3 is verbatim in
docs/reviews/upstream-issue-autotune-candidate-coin-flip.md — the GitHub
token in this setup is read-only, so it needs a token with repo scope.
| id |
defect |
fix |
| B1 |
53 files of vendor/dormouse-fused could not compile on 0.22 (TensorData::bytes is a method now), so none of those gates could run at all — the “gate that cannot compile reports nothing” class |
be86247 |
| B2 |
The production-shape gate could never pass: its fixture fed the state as [B,H,T,K] where the op’s contract is S [B,HV,K,V], and the header spent 11 days blaming the tensor-ops reference for the resulting shape panic |
a4f42ed — and the gate now checks the shipping arm against the loop arm: drift 3.3e-4, gradients ≤ 3.3e-4, i.e. the KDA chunk math is sound at our shape |
| B3 |
vendor/cubecl-fix (6.2 GB, untracked) resurrected after the v22 cut, masking any doc that still named it |
removed; doc-refs green |
| id |
defect |
evidence |
what it needs |
| C1 |
The corruption’s fingerprint is the engram table. Two identical processes, one clean and one corrupted, leave 4 of 54 parameters different; the biggest by five orders of magnitude is loop_block.engram.memory.embedding.weight — 10 063 of 32 768 rows, worst entry 4.2e18 absolute. Every KDA parameter is bit-identical |
ab_bisect in wt/bisect022 |
the line where 1e18 enters. Closed tonight: the gather’s backward (burn zeroes untouched rows exactly), the seeded RNG (sound on bare and autodiff devices, max 5.34), --engram-ram (also explodes 3/3) |
| C2 |
Our amplification. A sub-1e-6-class numerical difference becomes a 1e8 loss within one forward. Even a deterministic burn will bite us here; this is the KDA/gating overflow path AGENTS §2.3 already names as un-isolated |
the same runs |
a bisect inside the LoopBlock’s forward for the term that overflows |
| C3 |
No detector. A corrupted run looks like a slow arm, so a corrupted run reaches an A/B verdict |
all of it |
the two-process bitwise-ce replay (~one flag, no model code) |
| workaround |
effect |
verdict |
--engram-ram (host-RAM tables, CPU updates) |
worst ce 234 / 2.2e5 / 2.3e8 over 3 runs |
does not help |
persistent_memory = "disabled" |
10/10 → 1/3–3/3 explosions, zero step-time cost (198–207 ms/step, identical ce, measured on --no-kda) |
moves the coin; not shipped, because shipping it would hide the defect (ADR-0011) |
--opt adamw / --no-kda / --no-engram / n_experts=1 |
clean in 1 run each |
dodge, not a fix |
dormouse-pre4 stack |
the only measured-trustworthy numbers |
use it for any A/B that has to be believed |
- A1 — reproducibility of the candidate choice (pin, or a verified winner). This is the blocker: without it, 0.22 cannot produce a number a reader can trust.
- A2 — the persistent cache, which is the natural home for A1’s fix and does not run today.
- A4 — a statement on which bf16 path 0.22 took, so our tripwire and AGENTS §2.1 can stop lying in one direction or the other.
- A5 — nothing, until our reference step is better than its own noise; then re-ask.
- A3 — hygiene, whenever.
The question was “is that all of it?”. It was not. Sweeping every target of
every crate in the root workspace (CPU cell and cuda cell) found five more of
OUR gates dead on 0.22 — and, in the same sweep, two machine facts that the
port made false.
| id |
defect |
why it was red |
fix |
| B4 |
The root workspace’s own tests did not compile: TensorData::bytes is a method on 0.22 (2 sites in model_seam.rs, 1 in msa_gate1_flex.rs) — the same class as B1, one workspace over |
a gate that cannot compile reports nothing |
.bytes() |
| B5 |
gr_seam 3/3 red — its fixture inherited DormouseConfig::default(), which is deliberately the byteflow A/B winner net (schema.rs:809, use_byteflow = true), while mor_bce_weight defaults to 0.05, so validate refused loudly. The fixture never changed when the default flipped |
fixture older than the default it inherits |
use_byteflow: false in the fixture (it tests GR, not byteflow) |
| B6 |
bf16_compute_on_cpu_is_the_fp32_answer_and_says_so red under --features cuda — the “drop on non-CUDA” arm was guarded by #[cfg(feature = "cuda")], i.e. the BUILD, not the DEVICE, so a cuda build on the Flex device took the bf16 arm and moved 1.898 |
compile-time guard where the contract is a runtime device property |
device check (matches!(x.device().as_dispatch(), DispatchDevice::Cube(_))), the drop is now computed, not compiled in |
| B7 |
preset_exec 2 tests red — plain9m shipped with no COVERED/cost-list entry; the depth override assumed every preset reaches depth 2 (the depth-1 control cannot, and the setter refuses loudly); the Muon+ formula 4*n_experts + 3 assumed TSCT |
the preset landed; the gate did not follow |
three test-side generalizations, and the formula now reads use_tsct ? 4*n_experts + 2 : 0 plus the engram key projection (which is Muon-routed in BOTH topologies) |
And the honest negative that closes one of them: bf16_matmul_on_flex_is_the_fp32_answer
(pinned in param.rs) proves the raw bf16 op is honest on Flex bare and
through the autodiff bridge. So B6 is entirely ours — the guard was hiding a
path, not causing a wrong kernel.
| shape |
fp32 |
f16 |
bf16 |
| 5120x768x2048 |
0.94 ms / 17.19 |
2.06 ms / 7.83 |
0.44 ms / 36.84 |
| 5120x2048x8192 |
8.58 ms / 20.03 |
20.14 ms / 8.53 |
3.90 ms / 44.06 |
- ADR-0016 bug 2 is dead on 0.22. Both bf16 parity tests in
backend-parity
(bf16_matmul_cuda, bf16_gemm_cuda), #[ignore]d by that ADR, now PASS, and
bf16 matmul runs at 44.06 TFLOP/s — the cuBLAS ceiling, 5.2x 0.22’s own fp32.
A new gate arm (bf16_gemm_speed, bf16_gemm_speed_model_shape) carries the rate.
- The f16 tensor-core path is NOT alive on 0.22: 7.83-8.53 TFLOP/s, 4.7x below
the 39.75 TFLOP/s the cubecl-ir patch bought on pre.4 and 2.4x below 0.22’s own
fp32 — and STABLE across 4 runs (8.47-8.50), so it is not A1’s coin flip
choosing a slow candidate; it is the route.
- The cause of the f16 regression is ours:
ae80b53 cut every
[patch.crates-io] for the port on the stated premise “stock 0.11.0 covers
f16 sizing”. That premise is now measured false. The asymmetry (bf16 fast, f16
slow, one stack, one harness) is a cubecl 0.11 question worth the upstream
issue — and re-landing the cubecl-ir patch is our own candidate experiment.
- Upstream (A1-A5): unchanged, with one more cubecl question (the f16/bf16 asymmetry above).
- Ours, found and fixed (B1-B7): seven. Five of them were found by this sweep
and none of them except B1 was on the previous list.
- Ours, open (C1-C3): unchanged — the engram-table fingerprint, our amplification,
no detector.