Skip to content
dormouse

BROKEN or KNOWN-WRONG

3.3 Broken, open, or undocumented

Section titled “3.3 Broken, open, or undocumented”

  • 0.22.0 is NOT trainable: the corruption is the autotuner’s coin flip (2026-10-09, measured). fp32 b10 s512 on preset small destroys its own run 2 runs in 3 on burn 0.22.0 + cubecl 0.11.0: ce 5.57 → 1e8..1e16 → NaN at step 1. It is not the weights (at --lr 0, a no-op update, every parameter is bit-identical step 0 → step 1 while the step-1 forward reads 1.76e9, and the held-out eval on those same weights reads NaN), not the KDA chunk math (the production-shape gate is green: batched vs loop 3.3e-4, gradients ≤ 3.3e-4), and not a race or an out-of-bounds write (memcheck and racecheck both report 0). It is per-process: 50 of 125 autotune keys get a different winning candidate between two runs of one program, and two keys separate the outcomes 4/2 with winners whose medians are 1.6 % apart — a benchmark coin flip picks between kernels that disagree numerically. Everything that “fixes” a run (--opt adamw, --no-kda, --no-engram, n_experts=1, the persistent_memory setting: size-match 10/10, enabled 1/1, disabled 1/3-3/3, enforced 0/1) only moves the coin, and all cost the same 198-207 ms/step, so no config change is shipped. Until upstream answers (issue drafted, evidence in docs/reviews/2026-10-09-022-autotune-corruption.md), burn 0.22.0 is not fit for an A/B or a production run — a corrupted run looks like a slow arm — and dormouse-pre4 remains the stack that produces trustworthy numbers. The cheapest loud detector that exists today: replay a step in two processes and compare the bitwise ce.

  • The eval’s memory arm has no test. b3d6914 says it in its own message: the eval call site is inside a 1400-line function and is not callable from a test, so the half of the fix that mattered — passing the keys the training step passes — is carried by the code and by the engram= field, and nothing would fail if it regressed. The decode half does have one (decode_seam, decode_wiring, both on the CPU backend). The cheapest real gate is to assert that the eval line’s engram=<rows>/<arms> field is non-zero whenever use_engram is true, on a 2-batch CPU run.

  • Whether the attention arm trains is unverified. ANSWERED YES on the trainer’s backend, 2026-10-01 (d8fa449, tests/kda_param_grads_cuda.rs): all 11 KDA parameter groups receive non-zero, finite gradients on AdBal at trainer shape — CPU NdArray and CUDA agree bit-for-bit on the ops path; short-conv (a config the trainer has never run) got its first-ever gradient evidence. Falsification: one detach() at cuda_dispatch.rs:449 kills 8/11 groups, the test NAMES them (the three survivors are exactly the post-sever ops), restore is sha256-identical. §3.2’s second retraction now applies to HISTORY only, and the warm-step bwd share (~205 ms of ~480) is real work, not headroom. Two NEW findings came with the proof, both carried, neither fixed: (1) the fused adjoint is wrong and non-deterministic — under NoCheckpointing it disagrees with a central difference of its own forward by 4.5e-2..2.6e-1 on q/k/v, beta_proj, decay.b_alpha, while the three post-op parameters match an independent CPU autodiff at 2e-7; decay-path amax moves 7–27% between identical runs. The test asserts flow + prints the disagreement. (2) a fused forward launches on the trainer’s backend and its result is discarded — ADR-0019’s silent-fallback shape; fused kda=0/0 on the eval line means “not on this path”, not “the fast path is slow”. Both are burn-gdn2-owner items with file:line in docs/reviews/kda-gradflow-2026-09-30.md. The DM_GDN2_BWD_TRACE line was not release-visible; the numeric test replaces it — the trace could never have proven gradient arrival.

  • The A/B budget has not been re-costed. docs/protocols/AB-PROTOCOL.md still prices a 2k-step arm at 53 min, derived from ~1.6 s/step on a run whose attention backward did not execute. The 25.8 s/step batch-8 figure that would replace it has no committed log and no benches/history.tsv row, so the honest state is the cost of one A/B arm is currently unknown, not “2.7 GPU-h”.

  • Four silent-data-loss fixes with no gate (7adda92): a resume overwrote the best checkpoint because best_eval_bpb was a local; .best was a different model because the .ngram sidecar did not travel with the weights; inf was printed as a score; and the sidecar is read as None => h, i.e. freshly seeded rows, when absent. All four are fixed, none has a test, and the commit says so.

  • A commit message described a fix its diff did not carry. b3d6914’s message spends a paragraph on the eval-keys fix; the fix is in 7adda92. Anyone citing the fix from the message would cite the wrong commit, which is how a “verified” claim becomes unverifiable. Check the diff, not the message — and when a message and a diff disagree, the disagreement is a defect in its own right.

  • Two implementations of the optimizer policy — RESOLVED 2026-09-28 (831e3a0), and this entry outlived the fix. The path-marker copy in train/src/optim.rs is deleted; core/src/routing.rs is the live policy (optim.rs::optimizer_groups calls it) and there is no twin. What was left after the cut — a second GroupCounts and a second module walker — is gone too (docs/reviews/dedup-optimizer-2026-10-01.md).

  • The fused RMSNorm kernel — CORRECTED 2026-09-30, and the old sentence understated it by a factor. It does not merely “never engage on the trainer’s backend”: it never produced a number on ANY device in the life of this project. First launch on a bare CUDA tensor failed LLVM verification — Expected operand type llvm.ptr, but found builtin.integer — because Shared::new_slice was sized by a runtime value. Every kernel in this tree that runs sizes its shared memory from #[comptime] (burn-attnres/src/fused_attnres.rs:127, burn-gdn2/src/kernel/chunk_adjoint_cube.rs:61); this one did not. Fixed in 34c5631 with the same pattern and a single pub const THREADS. The reachability story is also sharper than the old wording: the deciding input is the device’s autodiff context (burn-dispatch-0.22.0-pre.4 src/tensor.rs:481-487 refuses the demotion unless DispatchAutodiffContext::Disabled), so a bare Device::cuda(0) runs the kernel and Device::cuda(0).autodiff() — every training forward and eval — does not. norm=0/N reproduced from the trainer’s own logs: 1560 asks per 500 steps, zero runs, 22 such readings across five logs, none with a non-zero first field.

  • WHAT THAT FIX IS NOT: gated. Reverting #[comptime] threads to a runtime value turns zero tests red, because no test runs the kernel — BURN_DEVICE appears nowhere under burn-rmsnorm and its Cargo.toml has zero required-features. So 34c5631 compiles and is verified by nothing. There is also no backward: rmsnorm_cuda returns a fresh Tensor::<2>::empty with raw handles written in, so the result carries no graph, and if the kernel were ever reached from an autodiff input the arm would return a LEAF, receive NO GRADIENT, and every loss curve would still look healthy — the 8fa5d4c defect verbatim. The dispatch guard is one ? wide. Until someone writes that gate, edits to fused.rs are verifiable only by hand.

  • 3 SILENT fallbacks remain in ADR-0019’s enumeration (41 sites total: 17 LOUD, 11 COUNTED, 13 SILENT, of which 10 have been fixed and 3 are proposals; the table lists every site by file:line). The notable ones: aux_loss returning None for a missing teacher latent, dspark_aux_loss contributing nothing when the sequence is shorter than the draft window, the pool install result being ignored, and a non-statted shard counting as 0 bytes in the size floor. A fourth, and the largest instance of the class in this project’s history, was found on 2026-09-28 and is fixed: the held-out eval passed hashed_ids = None (row 41, 7adda92). It survived this long because row 40 classified the None hash-key arm as “correct by construction — inference without hashed ids”, which is true for a decode path and false for a measurement. The correction, and the rule it generalises to, are in ADR-0019: a fallback is not excused by having a defensible meaning, it is excused by the caller being able to tell the reader which arm ran.

  • StressMonitor state is lost on resume (ADR-0021 item 4): the 201-entry window, the spike count and the p99.9 restart empty. It corrupts the report, not the objective. Fix: to_bytes/from_bytes plus a fourth container section or a <name>.stress sidecar, ~20 lines.

  • The .ngram sidecar carries no training-step stamp (ADR-0021 item 7): a crash between the model write and the sidecar write leaves a mismatched pair that nothing can detect. The fix is a train_step field in the header and a refusal to load a mismatched pair — deferred because it invalidates an 18 GB production artifact and that wants its own decision.

  • DSpark’s window was one position shifted — FIXED 2026-09-29, and it invalidates every DSpark number this project has ever recorded. The window was built from targets, the LABEL sequence, so with targets[q] == x[q+1] the draft head’s step s was fed x[p+s+1] — the very byte logits[p+s] had just predicted — and supervised toward x[p+s+2], while the hidden state it was handed, h[p+s], had only seen x[0..=p+s]. The head was handed the answer, one step early. dspark_aux_loss’s window arithmetic (aux.rs:263-282) was correct throughout and is unchanged; the fix is one line in model.rs (feed the teacher’s input_ids, gate still on targets.is_some() so a decode forward builds no aux graph). Gate: aux::tests::dspark_aux_does_not_read_the_label_sequence — the DSpark term is a function of (input_ids, hidden, logits) and of nothing else, so the label sequence must not move it; with the bug it reads 0.17736106 vs 0.17743118. No DSpark number survives it: every run with dspark_weight > 0 measured a different objective and trained its aux heads on it. Nothing in ~/logs/, benches/history.tsv or docs/ is a DSpark result, so there is no number to retract — but the next one is a first measurement, not a continuation.

  • dspark_stride is the ANCHOR SPACING of the DSpark window, in byte positions (the stream is bytes, so one position is one byte; not a token count, not a fraction). Windows start at p = 0, stride, 2·stride, … and the anchor count is floor((t − k − 1) / stride), so the first anchor needs t ≥ k + 1 + stride and a shorter sequence gets NO window and an exactly zero term. Default 16, which at k = 4 / seq_len = 512 is 31 anchors (p = 0 … 480) — i.e. the field is a sampling density (1 = a window at every position, 16 = ~6% of positions) and the term’s cost scales with the anchor count. 0 is refused loudly by config::validate. Why striding and not the paper’s random anchor sampling: sampling is a second RNG stream, and burn’s global RNG is never seeded — the same defect that forced the JEPA mask to be rebuilt from (seed, step) (ADR-0021). Two runs of one config drew different anchor sets, no A/B was reproducible and a resume changed the objective. A fixed stride is the same coverage per step with none of that; it is a determinism fix, and it has never been A/B’d against sampling, so “deterministic and documented” is the claim — not “equivalent to the paper’s estimator”. Gate: aux::tests::dspark_stride_is_the_anchor_spacing.

  • Adaptive depth is only an arm: --rand-depth (sample T per step) and MoR (rank the slots per position) both exist, are mutually exclusive by a loud refusal in resolve, and neither has been A/B’d against fixed depth.

  • --gen-max-iter does not exist; a flag help string advertises it. The CALM-lite confidence exit from ADR-0013 is not implemented either.

  • The vocabulary itself — docs/glossary.md ends with 20 code-vs-document disagreements: 6 fixed in this commit, 4 half-fixed (what remains is in README.md or a crates/ help string another owner holds), 10 open. The ones that still need a decision: three stale PonderNet references, the “CPU Adam” name for a Nesterov+Sinkhorn update, the small param count the memory budget is argued from (loop_block.rs:706, schema.rs:130), a retracted precision claim still living in a code comment (train/src/lib.rs:300-302), the two competing routing implementations, and the nano-fused preset named after a deleted module.