Skip to content
dormouse

Verdict audit, 2026-09-25

Doctrine applied to CODE (ADR-0002): every module must be held up by a recorded verdict, else it is a deletion candidate. Sources read in full: docs/architecture/PLAN.md, ADR-0001..0008, docs/archive/fused-verification-2026-09-23.md, AGENTS.md (GPU quirks), docs/research/2026-09-21-per-gb-sota.md, docs/archive/research/2026-09-22-fast-training-kernels.md, docs/archive/research/2026-09-23-fused-flagship50.md, docs/research/2026-09-25-nasa-burn-rust-practices.md, .bulba/memory.md, .bulba/plan.md, benches/history.tsv, git log, and every file under crates/ (branch fused-grad-coverage @ 9605896).

Meta-finding first: the enforcement instrument is dark. benches/history.tsv contains a header and ZERO rows. scripts/bench.sh exists. PLAN §“Performance budget” says a regression against the last entry blocks the merge — it has never been able to, because no entry was ever written. Every verdict below that rests on a wall-clock number rests on ad-hoc log reading, not the harness. Recording verdicts in docs while the canary TSV stays empty is the doctrine as paperwork.

1. Per-module table

Section titled “1. Per-module table”

LOC = source lines (wc). “Verdict” = the doc/measurement that justifies the file existing today.

File LOC Holding verdict Action
core/lib.rs 64 seam + gather_ce_matches_one_hot (CE swap 2026-09-04) KEEP. But fnv_hash (lib.rs:20) duplicates fnv (data/lib.rs:20) — same constants, two copies.
core/model.rs 345 ADR-0006 Keep (LoopBlock + fp32-head NaN rule, AGENTS) KEEP. max_seq_len() hardcodes 4096 (model.rs:343-345) while schema ships a max_seq_len field the runtime never reads — pick one.
core/loop_block.rs 375 ADR-0006 Keep; L_Rec-in-loop (4D-slice crash rule) KEEP, SIMPLIFY: forward_full_state is 235 lines (141-375). max_iter=4 verdict (PLAN 0b, 2.13x + iter8 NaN at step 193) is NOT applied: schema default is still 8 (config/schema.rs:63), small.toml still 8. A recorded verdict that lost to the code is a drift, not a verdict.
core/attention.rs 78 3:1 blend Keep (ADR-0006); MSA itself = “pending A/B” (ADR-0006) + long gate (ADR-0004) CONDITIONAL. MSA code carries a known engineering tax (gradient_detach leak, AGENTS) for an arm that has never won anything at s512. Survives until the long gate by explicit ADR; do not extend it.
core/aux.rs 229 JEPA 0.05 + DSpark: ADR-0006 Keep (external evidence, research §5); EMA no_grad fix: phase 0 verdict CONDITIONAL, split: KoLeo is a scheduled knife (PLAN phase 3 item 3; research §5: zero LM-scale evidence, DINOv2 origin) — aux.rs:28,84-85. The conf head’s own loss is discarded by destructuring (aux.rs:220 let (total, _ce, _tv, _conf)) — check whether total even contains it before claiming the head trains. No local confirm-tier A/B of aux-vs-pure-CE exists anywhere.
core/config/ (schema, loader, override, validation, mod) 420 ADR-0005 (seam + snapshot + drift check), tested KEEP, SIMPLIFY: 8 dead fields — ponder_w, rec_w, guard_w, keep_frac, dropout, rope_base have zero readers anywhere (grep: only schema/override/validation); halt_theta is copied into LoopBlock (loop_block.rs:52,130) and never read; max_seq_len is read only by loader tests while the model hardcodes 4096. Delete the fields + their override.rs:45-51 arms + validation.rs:14-16 checks.
core/gr.rs 117 NONE. gr_converges (train/lib.rs:1263) is a CPU convergence smoke, not an A/B. PLAN never schedules GR vs ReZero; ADR-0006 has no GR entry; AGENTS says “convergence-tested on NdArray”. CONDITIONAL — highest-priority missing verdict. GR is also gated OUT of fused and out of the flagship recipe, so it cannot even ride along. A/B or delete.
core/param.rs 133 pad-to-4 (AGENTS quirk); per-entry max_ortho (AGENTS 2026-09-04 ortho_probe) KEEP.
core/act_quant.rs 170 “VERIFIED, 100 steps, 0 NaN, convergence == fp32” (PLAN item 2) CONDITIONAL. Verified correctness ≠ BPB win; PLAN itself says the burn path “verifies correctness, not speed”. Its only payoff vehicle is fused/ (STE in-kernel). If fused/ dies, this module’s speed case dies with it — re-decide then.
core/fused/ (mod, backward, kernels) 4,944 ADR-0003 smoke (1.7-2.0x) SUPERSEDED: 1.25x SLOWER at flagship (fused-verification-2026-09-23.md criterion b; research 2026-09-23-fused-flagship50). Shelved pending direct-Cube adjoints; blocked on pre.4 (memory.md 2026-09-25: upstream works, OUR vendor port is the cuEventCreate-700 bug). CONDITIONAL — on its final gate. The doctrine already has its confirm-tier measurement and fused lost on step time; only the direct-adjoint hypothesis keeps it alive. See kill list #1.
core/fused/kernels.rs dead adjoints ~215 of 1553 kda_backward_cube (:1317), msa_backward_cube (:1376), engram_backward_cube (:1441) have zero callers (grep). The live arms backward routes through arms_inner_adjoint (backward.rs:1413) — the exact re-run-per-iteration mechanism the verification doc names as the cause of the 1.25x loss. The “escape hatch” the docs point to is unwired. Wire it or the next fused gate fails and everything goes.
core/fused/tests.rs 2,487 ADR-0003 + grad-coverage (22/22 CUDA green, fused-verification §Suite) KEEP exactly as long as fused/ lives — this harness is the only reason the 37 MB ckpt hole was caught.
core/examples/test_single.rs, test_ref_arms.rs, test_arms_grad.rs 85 none — scratch mains from the bisect sessions; fused/tests.rs covers the same ground with assertions DELETE. Phase 0 C3 already established the “tmp examples” pattern as deletable.
core/examples/bench_fused.rs, bench_fused_large.rs 176 superseded by the 50-step flagship A/B (research 2026-09-23) DELETE, or fold the surviving shape into scripts/bench.sh, which is the instrument that actually has a verdict contract.
core/examples/quant_probe.rs 80 AGENTS-maintained (up to date with the API) KEEP.
core/tests/model_seam.rs 524 phase 0 exit criterion (7 tests) KEEP.
data/lib.rs 372 2026-09-25 verdict (memory.md): pretrain v21 died on a silent b’x’ filler; now panics loudly + size floor (9605896). Ring/shuffle: phase 0. KEEP. Note next_batch_with_tables still space-pads short reads (lib.rs:349-351) — the loud assert in refill() covers the dry case; the pad is now provably unreachable dead tolerance. Consider an assert instead.
data/bin/filter.rs 1,427 phase 0 artifact: 46.2 GB → 19.7 GB, 272M → 104.2M docs, stats recorded (PLAN phase 0). Hand fastText inference matches dclm’s python predict path bit-for-bit by construction docs. KEEP until phase 2 closes (mixture sweeps re-run it). Phase 2’s confirm (filtered beats raw BPB) has NOT run — the corpus verdict is open; the tool that produced the corpus is not the thing on trial.
train/lib.rs 1,542 every mechanism inside has its own verdict (drift check ADR-0005; NaN halt; ortho fallback 2026-09-04; host-adam-every 2026-09-04) KEEP, SIMPLIFY: train_loop is 584 lines (445-1028) — P10 R4, see §3.
train/optim.rs 476 Muon+ recipe ADR-0006; validate_routing (stale-marker fail-loud); head-wise Q/K split (report §3.1) KEEP. ns=8 vs research item 7 (ns 8→5, Polar Express): recorded as hygiene, not done — cheap, do it with the next retrain. QK head-wise routing has routing tests but no local A/B — acceptable, it mirrors the report’s recipe and costs milliseconds.
train/offload.rs 376 Engram offload = best-evidenced component in the repo (research §3: SCONE + Engram + Over-Tokenized + Memory Layers). KEEP mechanism, flag the silent optimizer swap: the code trains tables with Nesterov + Sinkhorn (offload.rs:26-32,127-151, DeepSeek V4.1-Flash §2.5) while AGENTS.md, PLAN, the flag name (--host-adam-every), and the +2.3% measurement all still describe CPU Adam. The Adam→Nesterov swap and the Sinkhorn-every-4 balancing shipped with no recorded A/B and no doc update. This is exactly the “harmless mechanism accumulates” state ADR-0002 exists to prevent.
train/stress.rs 107 report §3.3 protocol; AGENTS doctrine: “stress protocol on every new config” KEEP.
train/jepa_targets.rs 135 motivated by the recorded batch-10 OOM (teacher = second full forward); offline-vs-live-teacher-vs-weight-0 A/B listed in research checklist #3, never run CONDITIONAL.
train/cfg.rs 205 ADR-0005 + tests; uncommitted diff extends PROGRESS_KEYS with eval/eval_every (numerics-free, consistent with the 2026-09-23 amendment) KEEP.
train/tests/ckpt_roundtrip.rs 104 resume math verdicts (2026-09-23) KEEP.
cli/train.rs 323 ADR-0005 typed-flag seam KEEP.
cli/serve.rs 202 the delivery contract (PLAN inference table: “$500-box offline chat demo is this table”) KEEP, but it is phase-5 code existing before phase-1 throughput — the doctrine’s step-time guardrail applies to it too: an OpenAI server nobody can serve from at 7 s/step is a liability until perf lands.
cli/generate.rs 45 smoke tool KEEP.

2. Ranked kill list

Section titled “2. Ranked kill list”
# Target LOC saved What would change my mind
1 fused/ entire (mod.rs + backward.rs + kernels.rs + tests.rs) ~7,900 One gate, already specified: wire the direct-Cube adjoints (kernels.rs:1317/1376/1441) seeded from saved forward buffers, re-run the 50-step flagship A/B. If fused is not < 7.05 s/step (burn’s measured number), ADR-0002 says delete — the 1.25x-slower confirm measurement already exists and the perf premise of ADR-0003 is retracted in writing (research 2026-09-23). The revival kit (pre.4 works upstream; our vendor fork is the bug, memory.md 2026-09-25) belongs to the fusion-backend plan (PLAN 2b), not to 6.5k LOC of hand adjoints.
2 Dead direct-adjoint kernels if fused survives instead ~215 Any caller appearing (the wire-up commit).
3 Scratch examples: test_single, test_ref_arms, test_arms_grad 85 Nothing; fused/tests.rs + model_seam.rs supersede them. Delete now.
4 bench_fused.rs + bench_fused_large.rs 176 Keep only if wired into scripts/bench.sh output; else delete.
5 KoLeo (aux.rs:28,84-85 + burn-jepa surface) ~15 + dep Its own scheduled A/B (PLAN phase 3 item 3). Prior: delete.
6 Dead config fields (ponder_w, rec_w, guard_w, keep_frac, dropout, rope_base, halt_theta, max_seq_len) + override/validation arms ~70 + snapshot surface A reader appearing for any of them. halt_theta in particular is PonderNet-halting remnants; the halting knife A/B (phase 3 item 2) decides both together.
7 burn-jepa/burn-dspark surface if the aux-vs-CE confirm ties both aux heads The confirm A/B (below). External evidence is fine-tune-scale (research §5); nobody has run the local 2k-step CE-only vs CE+aux pair.
8 max_seq_len hardcode OR field 3 or 1 Pick the field; generate.rs/serve.rs then honor presets (base=1024, one_b=2048) instead of silently using 4096.

3. P10 violations worth fixing (top 10, file:line)

Section titled “3. P10 violations worth fixing (top 10, file:line)”

Lens: docs/research/2026-09-25-nasa-burn-rust-practices.md rules 2/4/5/7/9.

  1. train_loop 584 lines — train/lib.rs:445-1028. R4 (≤60). Split: snapshot/drift, warmup, quant-check, the step body, eval, ckpt.
  2. ponder_loop_step 563 lines — fused/mod.rs:648-1210 (moot if kill list #1 executes).
  3. forward_full_state 235 lines — loop_block.rs:141-375. R4.
  4. ponder_loop_step_arms 340 lines — fused/mod.rs:1290-1629.
  5. arms_inner_adjoint 155 lines — fused/backward.rs:152-306.
  6. Assertion density ≈ 0 in the train path (R5 wants ≥2/function). loop_block.rs: 0 assertions in 375 lines. train/lib.rs step body (650-1022): 0 assert!/debug_assert! in 584 lines — invariants like hashes.len() == b*t*3, g_vec.len() == uniq.len()*dim are checked by silent if (lib.rs:875) that skips the update on mismatch instead of failing. That silent skip is the pretrain-v21 failure shape again, one layer up.
  7. Heap allocations in the step loop (research rule 4): offload.rs:132 (block vec per update), offload.rs:362-364 (rows Vec + from_data per step), train/lib.rs:874 (g_vec per host step), train/lib.rs:667 (pshift rebuilt per step), data/lib.rs:308-323 (hashes Vec per batch). All host-side and small, but they are the documented rule and trivially hoistable to init-sized buffers.
  8. Silent fallback chain in HostNgram::from_bytes — offload.rs:187-213: returns Option and the caller (train/lib.rs:516-528) prints one line and starts fresh tables on any mismatch. For a 37 GB sidecar, “v1 layout? — starting fresh” should be a hard error, not a warning; the v21 incident is the precedent.
  9. unsafe { std::slice::from_raw_parts } — fused/backward.rs:118-121, gated on an env var, not #[cfg(test)]/debug_assertions; a stray DM_FUSED_BWD_DEBUG=1 in a release run reads OOB-truncated buffers. Debug-only in intent, not in mechanism.
  10. Validating fields nothing reads — config/validation.rs:14-16 + override.rs:45-51 enforce ranges for ponder_w/rec_w/guard_w/keep_frac/dropout/rope_base (dead per §1). C-VALIDATE applied to dead knobs is worse than no validation: it manufactures confidence in a surface that does nothing. (Good news: no dyn in any dormouse hot path — enum dispatch (LinearLikeInner, ActFormat) and const-generic burn tensors everywhere; rule 8 is already met.)

4. Missing verdicts — A/Bs that must be scheduled

Section titled “4. Missing verdicts — A/Bs that must be scheduled”

Ordered by how guilty the code is without them:

  1. GR vs ReZero at matched budget, s512 confirm tier. gr.rs has never been A/B’d; it is excluded from the flagship and the fused path, so it accumulates as unvalidated surface. (The Qwen report’s −0.026 at 276B tokens is not a local verdict; ADR-0007 says every A/B is a vote on the flagship, not a substitute for one.)
  2. Aux (JEPA+DSpark+KoLeo) vs pure-CE at matched budget on the real corpus. All keep verdicts are external-paper citations; the only local aux measurement is stability (the EMA fix). Two flags make this a config-only confirm run.
  3. KoLeo removal — already scheduled (PLAN phase 3 item 3); fold into #2 as the third arm.
  4. Host-table optimizer: Nesterov+Sinkhorn vs Adam — the swap already happened in code (offload.rs) with no measurement; either record the verdict retroactively from a paired run or revert to the state the +2.3% number describes. Also: rename --host-adam-every or the flag lies.
  5. max_iter default: verdict EXISTS (4 replaces 8, PLAN 0b) — applying it to schema.rs:63 + configs/small.toml is not an A/B, it is executing a recorded one. Currently unexecuted.
  6. Filtered vs raw corpus confirm (phase 2 exit) — now a stability requirement, not a quality wish (PLAN 0b: raw is unlearnable at these step counts).
  7. PonderNet halting knife (phase 3 item 2) — halt_theta dead code and p_dist/KL machinery live on the hot path of every run pending this.
  8. TSCT vs plain experts at matched active params (phase 3 item 4); rank 64 vs 32 is the cheap inner A/B nobody has ever run — rank is a schema knob with zero recorded measurement, and it drives expert param count, retract cost, and GR’s bottleneck width.
  9. MSA at s512 (pending, ADR-0006) and the long gate (ADR-0004) — already scheduled; listed here because every confirm run until then pays MSA’s leak-tax for free.
  10. Offline JEPA targets vs live teacher vs jepa_weight 0 (research checklist #3) — jepa_targets.rs + the fused-path offline branch exist on faith.
  11. Fp8 factor-forward — AGENTS notes post-ortho-fix runs are the first real Fp8 A/B; not yet run. --quant fp32 remains the rollback.
  12. act-quant burn-path A/B — convergence-equal is not BPB-positive; decide together with fused/’s fate since the speed story lives there.

Verdict

Section titled “Verdict”

Adopt-with-changes, aimed at the repo not an idea: the doctrine is real but enforced asymmetrically — loudly for mechanisms (fused got measured and lost; max_iter got measured and won), silently for code (gr.rs, the offload optimizer swap, dead config fields, empty history.tsv all shipped without one). The single highest-leverage act is not a deletion, it is the fused gate: the direct-adjoint kernels are already written, unwired, and are the documented cause of the only confirm-tier loss on record. Wire them, run the 50-step A/B, and either fused earns its 7,900 LOC or the largest deletion in repo history is pre-justified by measurements that already exist.