Skip to content

The A/B seed wave — six arms × 3 seeds × 2k steps, and the three measurements that came with it

Lane: GPU queue dispatcher after the 100k run. Started 2026-10-01 13:30, on main at e638da7. Brief: serialize every CUDA consumer after the 100k production run, then land the seed wave of the six new arms against the night’s control family.

Every number here is BEST HELD-OUT bpb from a first_run_2000* log in ~/logs/, over the 81 920 B fixed eval window (20 batches × batch 8 × seq 512 = the window the whole control family shares, §2.6). Two runs with different batch do not share a window; every run in this file is batch 8.


0. THE CONTROL FAMILY, EXACTLY — and two corrections to the brief

Section titled “0. THE CONTROL FAMILY, EXACTLY — and two corrections to the brief”

The brief gave “the control finals were 6.314/6.329/6.387/6.396/6.437/6.443” (6 values), “control mean ≈ 6.37, spread ≈ 0.06”, and seeds 1001/1002/1003. All three need fixing, and the fixes change what the wave means.

0.1 There are SEVEN night logs, and they are TWO families, not one control

Section titled “0.1 There are SEVEN night logs, and they are TWO families, not one control”
log ckpt name aux retract_every seed BEST HELD-OUT bpb
first_run_2000_0930_1855.log first_run_2000 ON (jepa 0.05) 1 (default) 1 6.387
first_run_2000_0930_1921.log ...--jepa-weight-0---dspark-weight-0- OFF (pure CE) 1 1 6.437
first_run_2000_0930_1940.log ...--retract-every-4- ON 4 1 6.329
first_run_2000_1001_0037.log ...--seed-2- ON 1 2 6.314
first_run_2000_1001_0057.log ...--seed-3- ON 1 3 6.329
first_run_2000_1001_0121.log ...--seed-2---jepa-weight-0… OFF 1 2 6.443
first_run_2000_1001_0147.log ...--seed-3---jepa-weight-0… OFF 1 3 6.396

The bolded three are the aux-ON control family. The other four are the A0 (pure-CE) family, already judged in benches/history.tsv (control-3seed 6.387/6.314/6.329 mean 6.343 vs a0-pure-CE-3seed 6.437/6.443/6.396 mean 6.425, “aux arms EARN their place, 3/3 seeds”).

The brief’s 6.37 / 0.06 is the mean and sample-sd of all seven pooled, i.e. of two different models’ recipes averaged together. Pooled: mean 6.3764, sample sd 0.0532, range 0.1290. That is a number about the union of two families and it is the wrong yardstick for a wave that runs aux ON.

The yardstick for this wave is the aux-ON 3-seed control:

n mean sd (sample) range (max−min) min max
control, aux ON 3 6.3433 0.0386 0.0730 6.314 6.387

§1.2’s bar is 0.0730 BPB — the control’s own seed range — and a win has to beat it, not beat its own sd. Both numbers are in the table so a reader can apply either reading.

0.2 The seeds are 1/2/3, and “1001/1002/1003” is a filename

Section titled “0.2 The seeds are 1/2/3, and “1001/1002/1003” is a filename”

first_run.sh’s log pattern is first_run_${STEPS}_$(date +%m%d_%H%M).log. first_run_2000_1001_0037.log is Oct 01, 00:37 — 1001 is the date and 0037 the time, not a seed. The seeds on record are 1, 2, 3 (from the ckpt names --seed-2-, --seed-3- and the seed = 1 in the config snapshot).

So the wave runs seeds 1/2/3, paired against the control’s seeds 1/2/3. This is deliberate and it is the better design: --seed is a pure function of model init and the JEPA mask (§3.7), so a same-seed pair starts from the same initialisation to within the measured TSCT residue (≤1.04e-06 relative), which removes the init term from the comparison instead of adding a second variance source. Running seeds 1001/1002/1003 would have compared against a control measured on 1/2/3 and charged the difference to the arm.

0.3 The recipe, read off checkpoints/first_run_2000--retract-every-4-.config.toml

Section titled “0.3 The recipe, read off checkpoints/first_run_2000--retract-every-4-.config.toml”

--preset small --batch 8 --seq-len 512 --no-engram --retract-every 4 --steps 2000 --eval-every 500 --guard --detach --timers --seed <1|2|3>, aux at the preset defaults (jepa 0.05 + KoLeo, dspark 0.0), eval_batches 20, lr 1e-4 cosine, opt mix (Muon+ ColRow ns=8), quant auto → Fp8 on sm_120, max_iter 4, rank 64, n_experts 3, d_model 768, d_ffn 2048, params 9 197 454. Launched by tools/first_run.sh — the control family was launched by that same script, and the ckpt-name formula reproduces its names exactly (--retract-every 4 → first_run_2000--retract-every-4-, which is how the trailing - on every one of those names is accounted for: tr -c maps the newline).

A confound I am naming rather than hiding: the 3-seed control family was run at retract_every 1, and this wave runs retract_every 4 (the owner’s working mode, and the mode the 100k is on). The cadence effect is measured at one seed only: seed 1 gives 6.387 (retract 1) vs 6.329 (retract 4), a paired −0.058. So the arm family is compared against a 3-seed retract-1 control plus a single retract-4 point. If the wave finishes with time in hand, two retract-4 control seeds (2/3) close this; if it does not, §7 lists it as the open precision gap.


1. The queue as briefed, and the two places I departed from it

Section titled “1. The queue as briefed, and the two places I departed from it”

Brief order: (1) wait for the 100k + record + board, (2) the two-chunk adjoint CUDA gate in wt/kda-adjoint, (3) generation from the 100k export, (4) the 6-arm × 3-seed wave, (5) per-arm verdict vs the control spread.

Two changes, both forced and both recorded here rather than silently taken:

  • Item 2 runs from main, not wt/kda-adjoint. The brief says the lane’s worktree. But wt/kda-adjoint has no target/, and a cold build of one test binary in it is the 686-package / ~41 GB / ~23 min figure the build lock was written for — against 29 GB free on /home at 94 %. It does not fit, and starting it risks a disk-full mid-link (§2.5: “a mold error that says ‘Disk full?’ is the disk”). Instead: diff -rq of vendor/dormouse-fused/crates/burn-gdn2 between main and wt/kda-adjoint reports no differences at all except two __pycache__ .pyc files. The lane’s test landed on main as 439d8c8 (the same change as its c3f5c31), and main has a warm 98 GB target/debug. So the identical gate runs from main for the price of one target’s link.
  • Item 2 is placed after item 3. Item 2 costs a build (~5-15 min) that item 3 does not, and item 4 is the owner’s gate #1 (“all arms A/B’d at 3 seeds”) and needs ~6.3 h of a window that has ~6.5 h left when the 100k exits. Spending 40 min of GPU-serial time on a diagnostic before the deliverable would cost one arm. Item 2 therefore runs immediately after the first arm’s first seed (its build overlaps nothing, and it is bounded at ~2 min of CUDA).

Everything else is as briefed: one CUDA process at a time, pgrep preflight before every launch, all builds through tools/build_lock.sh run abwave --, logs in ~/logs/, sequential arms, no push, no deletions, no config-number changes.


2. The six arms, and what each one’s flag is

Section titled “2. The six arms, and what each one’s flag is”

Flags read from the arms’ own landing reports and docs/protocols/AB-PROTOCOL.md, not guessed. Every arm is off by default for checkpoint compatibility, and each is a single --set into the resolved config (--set is a generic serde override, config/override.rs:11, so every schema field is reachable without a typed flag).

# arm flag(s) source param delta on small what proves it engaged
1 mHC (manifold-constrained hyper-connections) --set use_mhc=true mhc-2026-09-30.md (row 7b), schema.rs:123 +0.067 % at the default mhc_streams = 2 params= on the log header line moves; probe::MHC exists but is not printed
2 SiTU-GLU (Kimi K3 Eq 12) --set use_situ=true situ-2026-09-30.md, AB row 9, schema.rs:140 +393 216 (+4.28 %) params=9197454 → 9590670 in the header
3 RoPE on the KDA q/k --set use_kda_rope=true AB row 8 — see §2.1 — the field does not exist on main
4 MoE routing --set moe_topk=1 moe-routing-2026-10-01.md, schema.rs (d_moe_topk() = 0) 0 (param-neutral by construction) the resolved-config snapshot + moe_lb_coef left at 0.0; see §2.2
5 AttnRes --set use_attnres=true AB row 7, schema.rs:122 the arm’s d_model-wide query vectors params= moves; probe::ATTNRES exists but is not printed (already owed, AGENTS §2.2)
6 future-byte head --set aux_fb_weight=0.1 AB row 1b, future-byte-2026-09-30.md +197 120 (+2.1 %) params=9197454 → 9394574 and the log line’s fb=<ran>/<asked>, the only arm counter the line actually prints

2.1 RoPE — the arm is not reachable from main, and that is a fact, not a bug to debug here

Section titled “2.1 RoPE — the arm is not reachable from main, and that is a fact, not a bug to debug here”

rope-2026-09-30.md:275-277 says so in its own words: “Not wired to the trainer’s config or CLI. The flag is on KdaConfig … wiring it into dormouse-core’s DormouseConfig and a CLI flag is the owner’s call about §5.” grep -i rope over schema.rs and cfg.rs on main returns nothing model-shaped. So --set use_kda_rope=true on main is an unknown config key, which config/override.rs refuses loudly. Per the brief’s own rule (“if an arm refuses to run, record the refusal LOUDLY and skip”) this is recorded as a refusal, and no code is touched: wiring a flag is a code change to another lane’s crate, not a dispatcher action. It is also the one row the protocol says to run as a post-training gate, not a BPB race (AB row 8), so it was never going to be decided by this wave.

The arm’s own fixture and its headline number are 4 experts / top-1 (moe-routing-2026-10-01.md:1, “dense shared FFN vs 4× top-1 at equal ACTIVE FFN FLOPs”), and small ships n_experts = 3. Raising n_experts to 4 to match would add a fourth expert’s worth of parameters and turn a param-neutral routing A/B into a capacity A/B, so the wave runs the shipped n_experts = 3 with --set moe_topk=1: top-1 of 3, still param-neutral (num_params equal at moe_topk 0 and 1 is the crate’s own gate), which is the property that makes it comparable to the control at all. moe_lb_coef stays 0.0 — the report ships no coefficient (“no coefficient is shipped, and moe_lb_coef stays 0.0 with the arm off”), and lb > 0 with routing off is refused.

Stated limit on this arm’s evidence: routing is the one arm here whose on-position does not change params=, and probe::MOE_ROUTE is not on the log line (only fb= and the legacy seams are). So for MoE, “the arm ran” is carried by the resolved-config snapshot plus the crate’s own off_is_the_dense_blend_bit_for_bit gate, not by a runtime counter. A silently un-engaged MoE run would be indistinguishable from the control in the log. That is a §1.1 (SILENT) gap in the log line, not in the model — listed in §7.


It finished at 14:42 on its own, with nothing done to it. Final line, verbatim:

done steps=100000 best ce=2.415 | BEST HELD-OUT bpb=5.545 at step 86000 -> checkpoints/first_run_100000--retract-every-4-.best.bin
steps 100 000, complete (planned 100 000)
wall clock 2026-10-01 02:02 → 14:42 = 12 h 40 m, ~456 ms/step averaged; warm steps read 463-515 ms
BEST HELD-OUT bpb 5.5452847 at step 86 000 (sidecar ….best.bpb, agrees with the log line)
final eval 5.729 at step 99 500
best train ce 2.415
NaN / firewall / panic / guard-restart 0 lines of all four in 2 126 log lines
window 81 920 B fixed, 200 evals
commit the release binary was built 09-30 18:31, off a7c3cad — which is before all six arm landings, so this run provably contains none of them
board python3 tools/trainboard.py --out /tmp/opencode/board-final ~/logs/first_run_100000_1001_0202.log → 4 PNGs + index, regenerated 14:44
history one row appended to benches/history.tsv (2026-10-01T11:43:00Z, commit a7c3cad)

Reading it honestly. The curve descends 6.480 @ 500 → 6.042 @ 40 500 → 5.769 @ 50 500 → 5.720 @ 75 500 → 5.603 @ 85 500 → 5.545 @ 86 000, and then it does not descend again: 5.595, 5.603, 5.666, 5.692, 5.740 for the last 14 k steps. So the number is best-of-a-plateau, not a late win, and the last 14 k steps (11 % of the run, 1 h 45 m) bought nothing — that is the same shape as §3.1’s 4.997 row, and the same honest reading applies. Against the 2k controls (6.343 aux-ON / 6.425 pure-CE, §0.1) the 100k run is 0.80 BPB better, so the budget itself is worth ~0.8 BPB over 50× the steps.

The board’s second trace makes the regime visible: at step 100 000 the train BPB is ~3.7 and the held-out BPB is 5.73. A 2.0-BPB train/held-out gap on a held-out set of 81 920 B is §2.3’s overfit class, not a measurement artefact — which is also why the last 14 k steps are flat: the model is spending them fitting bytes it has already seen. The next budget question is therefore not “more steps” (the curve says no) but “more distinct bytes” (queue row 3, working set) or a larger model (the owner’s ×10 gate, .bulba/goal.md).

Two counter readings from the same log, because they are on the line and a reader will trip over them:

  • fused kda=398012/0 bwd=0 … declined=2037796 ops=811936 node_bwd=0. The fused forward ran 398 012 times and its result is discarded; the gradient path was the ops arm every step. That is §3.3’s FINDING 2 (ADR-0019’s silent shape: a run that launches a kernel and throws the answer away), and this run is another instance of it, not a new defect.
  • norm=0/911936: the fused RMSNorm kernel was asked 911 936 times and ran zero — the reachability story of §3.3, unchanged, on a 100k-step run.

4. The two-chunk fused-adjoint gate on CUDA — GREEN, and deterministic

Section titled “4. The two-chunk fused-adjoint gate on CUDA — GREEN, and deterministic”

Run 2026-10-01 15:21-15:27, card otherwise idle (checked immediately before each launch), test result: ok. 1 passed; 0 failed, 3.2-5.7 s per run.

How it was run, and why not in wt/kda-adjoint. The brief said to run it there; §1 explains why that was not possible (no target/, 41 GB needed, 23 GB free on /home). What ran instead was the same test binary the lane had already built:

/mnt/e43497ab-0ff2-45b4-b45f-28de3339a53e/kdagf-target/debug/deps/fused_adjoint_f64-26e48d2e1534219b
LD_LIBRARY_PATH=<main>/target/debug/build/openblas-src-*/out/OpenBLAS-0.3.32
--nocapture --test-threads=1

Three provenance facts make that the same measurement, not a shortcut around it: diff -rq of vendor/dormouse-fused/crates/burn-gdn2 between main and wt/kda-adjoint reports no differences (only two __pycache__ .pyc); the test file’s mtime in that worktree is 11:38:53, the binary’s is 12:07:49, so the binary post-dates the two-chunk edit; and the run prints which scope it is in, so the binary proves its own vintage — both === ONE CHUNK (BK1 only) === and === TWO CHUNKS (BK1 + BK2 + the d_s_shift glue) === appear. The lane’s LD_LIBRARY_PATH was needed because openblas-src emits its .so into the build dir with no rpath; the file lives in this worktree’s target, which is where the lane got it from too.

4.1 The numbers (bar: 1e-3 relative on the f32 side; oracle self-consistency ~1e-12)

Section titled “4.1 The numbers (bar: 1e-3 relative on the f32 side; oracle self-consistency ~1e-12)”
input ONE CHUNK (BK1) TWO CHUNKS (BK1 + BK2 + d_s_shift)
fused forward vs the f64 transcription rel 2.381e-7 rel 2.094e-7
d_q 2.416e-7 2.605e-7
d_k 1.642e-7 1.771e-7
d_v 2.419e-7 3.713e-7
d_g 2.496e-7 4.410e-7
d_b 1.402e-7 2.044e-7
d_w 2.329e-7 2.248e-7
d_state 2.655e-7 1.782e-7

Every one of the fourteen cells is ok, and the two-chunk arm is 1.4-2.5× tighter than its bar, not near it.

4.2 Six runs, because “green once” cannot decide a race

Section titled “4.2 Six runs, because “green once” cannot decide a race”

The gate costs 3-6 s, so it was run six times. All six: 1 passed; 0 failed, and all seven two-chunk maxima identical to 7 significant figures (q=2.605e-7 k=1.771e-7 v=3.713e-7 g=4.410e-7 b=2.044e-7 w=2.248e-7 state=1.782e-7), as were all seven one-chunk maxima.

4.3 The verdict, and the three things it is not

Section titled “4.3 The verdict, and the three things it is not”

GREEN and reproducible at the two-chunk scope. BK2 — the sequential reverse recurrence over chunks, plus the d_k_bptt / d_e_bptt / d_s_shift glue that 2a430cc measured at rel 3.3e-1 and 6.2e-1 before its fix — matches an independent f64 oracle to 4.4e-7 at worst, and does not move across runs. So on the reading docs/reviews/kda-adjoint-2026-10-01.md §4 offers as (a) — f32 conditioning of the instrument, not a race in the kernel — the evidence lands on (a): a term that is ~1e-4 of |L| is exactly the magnitude this oracle resolves to 4e-7, so an f32 finite difference of it moving 7-27 % is the instrument’s error budget, not the kernel’s.

Stated limits, so nobody reads this as closing more than it closes:

  1. Scope is 2 chunks (T=32), and d8fa449’s disagreement is at 16 chunks (T=256). Different scope, and the 16-chunk disagreement is not resolved by this run — it is only shown to be a scope question, not a broken BK2.
  2. The instrument is an f64 oracle, not a finite difference. So this run does not speak to the precision of the FD instrument itself; it speaks to the kernel.
  3. Determinism is measured at the seven printed maxima, i.e. to 7 significant figures of an aggregate. Identical aggregates are strong evidence and not a bitwise proof of every output element.
  4. BK2-gated does not mean BK2-enabled. §6 of that review is unchanged: on the trainer’s backend the fused op declines, so the ops path still carries the gradient, and nothing here argues for turning the kernel on.

(§4’s verdict is also the answer to the brief’s premise that this run “does not exist”: a run of exactly this binary was on the card at 14:43, ~20 min before I re-ran it myself, and my numbers are independent of whatever that one printed.)


5. Generation from the 100k export vs the 2k export

Section titled “5. Generation from the 100k export vs the 2k export”

Run 2026-10-01 15:32-15:40 with ./target/release/generate, the binary rebuilt from main at 15:19.

5.1 The first thing to say: it is not the 100k model

Section titled “5.1 The first thing to say: it is not the 100k model”

--export checkpoints/first_run_100000--retract-every-4-.f32.dmexp prints loaded … (F32 weights, 9197454 params, **step 75000**). The export was written at 11:20 from a step-75 000 checkpoint, so every sample below is a step-75 000 model, not the finished run (whose best held-out point is 5.545 at step 86 000, §3). The 2k export is likewise step 1500, its own best point. So the comparison is 75k-steps-vs-1.5k-steps of held-out BPB 5.72 vs 6.39, which is the honest framing of every row below.

$ ./target/release/generate --export … --prompt The --steps 120 --temp 0
cannot sample empty range

The sampler computes exp((x - max) / temp); at temp = 0 the winning element is 0/0 = NaN, the sum is NaN, and gen_range refuses the range. So the brief’s “temperature 0 (greedy)” cell is not available in this binary, and the honest thing is to record the panic rather than substitute a near-zero temperature and call it greedy. Greedy decode is a one-line branch (argmax when temp <= 0) in crates/dormouse-cli/src/bin/generate.rs:47; it is the inference owner’s call, not this lane’s, and it is listed in §7.

5.3 What the models actually emit, counted

Section titled “5.3 What the models actually emit, counted”

Prompt The, 150-300 steps, same binary, same export format. “real” = a token in a 100-word common-English list; digits/upper/punct = how much of the output is symbol soup.

temp model bytes tokens real words distinct digits uppercase punct
0.15 100k, step 75 000 153 32 12 — 37.5 % 24 0 1 0
0.15 2k, step 1 500 146 25 4 — 16.0 % 24 1 2 2
0.30 100k, step 75 000 147 15 2 — 13.3 % 15 10 6 6
0.30 2k, step 1 500 152 22 2 — 9.1 % 22 15 15 15
0.80 100k, step 75 000 221 43 5 — 11.6 % 37 39 47 17
0.80 2k, step 1 500 283 56 1 — 1.8 % 44 33 70 29

At temp 0.15, verbatim:

100k: Thers o the woreronated the the thicecor t the to thartonds tor the the isthe
cons on to tho the alor intritow bo toler s or cof cutiron fero bu the thet
2k : Themanseve th us s anteg sengonatorsintwe s te Lis en tere melemisons
benegit f es wige ( meatiollinsgelere of ty on as ged m1 \( imiorosome betho

The 100k sample contains 8 occurrences of “the” and no digits and no uppercase; the 2k sample contains zero and drifts into (, \ and symbol soup. So the trained model has real English function-word structure where the 2k model has none.

Verdict, honestly bounded:

  • At the archive’s temperature (0.8) the comparison says nothing about either model. Both are noise; a model at 2.3 bits/byte has no readable 0.8-temperature sample to give. The 100k number is nominally better (11.6 % vs 1.8 %) and that ordering does survive at 0.15 and 0.30 — but the absolute quality is “fragments”.
  • The brief’s baseline is not reproducible as described. I could not get the 2k model to emit anything like “The the the in con ate” at any of the three temperatures; at 0.8 its output has no repeated “the” at all. Either that sample came from a different temperature, a different build, or a different prompt. I am not claiming to have reproduced it, and the comparison above is against a freshly generated 2k sample, not against that string.
  • n = 1 sample per cell. These are counts on one 150-byte continuation each. They rank the two models consistently across three temperatures; they are not a statistic and must not be quoted as one.
  • A byte-level 9.2M model at step 75 000 is not a text model yet. The honest summary is that 100k steps bought function-word structure that 2k steps did not have — visible in the counts, invisible in anything you would want to read.

6.1 The finding that reshaped the queue: four of the six arms could not be set

Section titled “6.1 The finding that reshaped the queue: four of the six arms could not be set”

The first launch attempt was --set use_mhc=true, the invocation docs/reviews/mhc-2026-09-30.md and AB row 7b document. It died in 32 seconds:

train failed: unknown config key "use_mhc"

--set is not a serde override. It is a hand-written match (crates/dormouse-core/src/config/override.rs:28-71) listing 33 keys, and use_mhc, mhc_streams, use_situ and use_attnres are not on it. The config fields exist in schema.rs and the arms are wired into the model; the trainer’s seam that turns a command-line string into a config value was never extended for them. So the four documented invocations (--set use_mhc=true, --set use_situ=true, --set use_attnres=true, --set use_kda_rope=true) have never reached a trainer process. Two of the six arms’ flags are on the list — moe_topk, aux_fb_weight — which is why this went unnoticed for a week: the two arms that were documented as runnable were the two that worked.

This is the ADR-0011 LOUD class doing its job (the refusal named the cause and the escape), and §1.7’s document-vs-code disagreement class. The fix is six lines in override.rs; it is not this lane’s to make (§1.6: do not edit files another lane is in), so it is a follow-up in §7 with the file:line.

The reachable seam, used instead: loader.rs’s is_explicit() accepts any string containing / or ending .toml as an explicit path, and DormouseConfig deserializes from TOML with #[serde(default)] on every field. So /tmp/opencode/presets/<arm>.toml — configs/small.toml verbatim plus one arm line, verified by diff — reaches all five arms with no code change:

$ diff <(grep -v '^#' /tmp/opencode/presets/mhc.toml) <(grep -v '^#' configs/small.toml)
31,32d30
< use_mhc = true
< mhc_streams = 2

Every arm’s preset is configs/small.toml plus one or two lines and nothing else changed, and no preset file was added to configs/ (the owner’s directory).

arm flag as documented reachable? how it was run
mHC --set use_mhc=true no (not in override.rs) presets/mhc.toml: use_mhc = true, mhc_streams = 2
SiTU --set use_situ=true no presets/situ.toml: use_situ = true
AttnRes --set use_attnres=true no presets/attnres.toml: use_attnres = true
MoE --set moe_topk=1 yes presets/moe.toml: moe_topk = 1 (moe_lb_coef stays 0.0)
future-byte --set aux_fb_weight=0.1 yes presets/fb.toml: aux_fb_weight = 0.1 (horizon 2)
RoPE --set use_kda_rope=true no field at all REFUSED — no schema field, no preset, nothing to run

(per-arm rows land here as runs complete; the yardstick is §0.1: control aux-ON mean 6.3433, §1.2 bar = the control’s own seed range 0.0730)

Every run below: --preset = configs/small.toml + the arm line(s), batch 8, seq 512, --retract-every 4, 2000 steps, seeds 1/2/3, aux jepa 0.05 + KoLeo, window 81 920 B, 0 NaN, 0 panics, BEST HELD-OUT bpb from the log’s own last line.

mHC (use_mhc = true, mhc_streams = 2) — TIE

Section titled “mHC (use_mhc = true, mhc_streams = 2) — TIE”
seed arm control (same seed) paired Δ
1 6.4130 6.387 +0.026
2 6.3130 6.314 −0.001
3 6.3700 6.329 +0.041
mean 6.3653 (sd 0.0502, range 0.100) 6.3433 (sd 0.0386, range 0.0730) +0.0220

Verdict: TIE. |Δ| = 0.0220 against a bar of 0.0730 — the arm is inside the control’s own seed spread, and §1.2 says a tie deletes the mechanism. Two things make this verdict stronger than a bare number:

  • The arm engaged, so the tie is a measurement and not a no-op: params=9203609 in every log against the control’s 9197454, i.e. +6155 params = +0.067 %, which is exactly the cost mhc-2026-09-30.md quotes for n=2. Same number, independently.
  • The report predicted this outcome from the mechanism, not from the curve: at the φ-lever’s left init the composite sits at the identity, so the arm starts from the same function as ReZero and differs in the operator — and the owner’s own note on it already reads “пока не внедрять в дефолт”. This wave turns that expectation into a 3-seed tie. mhc_streams = 4 (the base paper’s own rung) remains the follow-up row the report names, and this result says nothing about it.

Against the exact-recipe control (the one paired --retract-every 4 seed-1 point, 6.329) the delta is +0.084, i.e. outside the bar in the losing direction — but that is a single seed and cannot carry a verdict, which is why §0.3 names the two missing retract-4 control seeds as this wave’s precision gap.

AttnRes and future-byte — run by the wave3 lane, NOT by this one

Section titled “AttnRes and future-byte — run by the wave3 lane, NOT by this one”

This dispatcher’s queue died before these two landed; the runs are in benches/history.tsv at 2026-10-02T04:25:00Z (commit af9def1) and are summarised here rather than re-run, because re-running them would spend card time to reproduce numbers that are already on disk with the same window, the same bar and the same verdict rule. Full rows: wave3-attnres-3seed-FRESH, wave3-fb-3seed-FRESH.

arm seeds mean vs control 6.3433 verdict
AttnRes (use_attnres) 6.191 / 6.346 / 6.453 6.3300 −0.013 TIE → deleted (§1.2); its own seed range is 0.262 = 3.6× the control’s 0.073
future-byte (aux_fb_weight = 0.1) 6.406 / 6.391 / 6.463 6.4200 +0.077 TIE → deleted (§1.2); widest loss of the wave, and its train CE (2.998-3.040) is the worst of any completed arm

Both arms verified engaged from the log: AttnRes params=9200526 (+3072), future-byte fb=1503/1503 and params=9394318 (+196 864). The AttnRes seed spread is itself the finding: a 0.262 range on three seeds of the same recipe means an AttnRes “win” of 0.05 BPB would have been noise, and it is 3.6× the control’s spread, so §1.2’s bar is not even the right width for that arm.

arm why there is no number
SiTU all three of this lane’s runs died at step 0. Same signature four times (see §6.5): the process goes single-threaded at ~97 % CPU, makes zero syscalls and zero I/O, and its GPU memory drops from ~9 GB to 699 MiB. It is a spin, not training. One run (attnres seed 2) got to step 200 before dying the same way. The cause was never isolated — the three days since have no core dump, no strace (not installed), and no surviving process to inspect.
MoE this lane’s own fault, and the mechanism is worth recording: my driver killed all three runs at 45 s with “ARM DID NOT ENGAGE”, because it compared params= against the control’s count. MoE is param-neutral by construction (num_params equal at moe_topk 0 and 1 is the crate’s own gate), so a param check cannot detect it — I killed three correct runs with a check that was only ever valid for arms that add parameters. The MoE lane later shipped tools/moe_ab.sh with its own preflight and recorded “ready, NOT run: no GPU, night2e owned the card all lane, so NO run and NO A/B result” (benches/history.tsv, 2026-10-03). MoE still has no A/B number.
RoPE no schema field. --set use_kda_rope=true has nothing to set, and docs/reviews/rope-2026-09-30.md:275-277 says wiring it to the trainer’s config is “the owner’s call”. Not run, not runnable.

6.5 The step-0 death signature (four runs, unexplained)

Section titled “6.5 The step-0 death signature (four runs, unexplained)”

The four runs that died at step 0 (situ seeds 1/2/3, attnres seed 1) all left a log of exactly 8 lines ending ckpt … saved step 0, and a process in this state:

elapsed 31:25 cpu 00:30:33 97.2 %CPU 1 thread RSS 5.8 GB State: R
/proc/PID/io: rchar and wchar FROZEN over a 10 s sample (zero I/O)
/proc/PID/syscall: 0 (no syscall in flight)
nvidia-smi: utilisation 6 %, 699 MiB used (a training step uses ~9 GB)

One thread, no syscalls, no I/O, GPU memory released: a userspace spin that has already given the device back. It is not an OOM (the ram-guard log’s last entry is 2026-09-30, and MemoryMax=40G never tripped), not a NaN firewall (the log would say so), and not a disk-full kill (that signature appears in the archive as ENOSPC, from the 02:05 event of 2026-10-02, and the disk had 70 GB free when these died). It is an open defect, not a result. The cheapest next step is one run under --memlog with DM_QUANT_DEBUG=1, or a core dump on the spin, and it belongs to whoever owns the trainer’s step boundary.

# arm seeds mean held-out BPB Δ vs control 6.3433 §1.2 verdict
1 mHC 3 (this lane) 6.3653 +0.022 TIE → deleted
2 SiTU 3 attempted, 0 completed — — no result (step-0 spin, §6.5)
3 RoPE not runnable — — no result (no schema field)
4 MoE 3 attempted, 0 completed — — no result (killed by my own check; the MoE lane’s own run also never happened)
5 AttnRes 3 (wave3) 6.3300 −0.013 TIE → deleted
6 future-byte 3 (wave3) 6.4200 +0.077 TIE → deleted

Three of six arms are judged, and all three are ties, which under §1.2 means all three are deleted. None of the three that reached a number earned its place: the best of them (AttnRes, −0.013) is a sixth of the control’s own seed range, and the two with the largest sample spread are the two whose “signal” would have been most likely to be noise. Three arms remain unmeasured, and one of those (MoE) is the arm the queue’s own §3.5 playbook cares most about at 1B.

That is the honest shape of this wave: it did not find a winner, and the mechanism it was built to find is now three deletions and three open questions. The cheapest next card-time is SiTU and MoE at 3 seeds each (~2.5 h), which is also the only way the “arms hurt at 9.2M” reading in benches/history.tsv:4c7721f (byteflow 3.466 / plain9m 3.947 / dormouse 6.498) gets tested on the arms rather than on the whole architecture.


7. What is NOT claimed, and the follow-ups this lane owes

Section titled “7. What is NOT claimed, and the follow-ups this lane owes”

7.1 Three defects this lane found, all in the seam the A/B queue runs through

Section titled “7.1 Three defects this lane found, all in the seam the A/B queue runs through”
  1. --set could not reach four of the six arms (§6.1) — FIXED, 2026-10-01 21:24, 68cb247 “fix(core): –set knows the four queue arms it loudly refused”, 2 h 40 min after this lane reported it. At the time of writing this section override.rs was a hand-written match over 33 keys with use_mhc, mhc_streams, use_situ and use_attnres absent from it, so --set use_mhc=true, --set use_situ=true and --set use_attnres=true had never reached a trainer; the documented invocations are live now. This wave therefore ran its arms through preset files, which is why its own logs name /tmp/opencode/presets/*.toml rather than a flag — that path is still the right one for any arm that adds a field the seam does not know.
  2. generate has no greedy decode (§5.2): --temp 0 divides by zero, produces NaN, and dies with cannot sample empty range. crates/dormouse-cli/src/bin/generate.rs:47 — argmax when temp <= 0 is the whole fix.
  3. The log line prints one arm’s counter. fb=<ran>/<asked> is there, and probe.rs counts MHC, SITU, MOE_ROUTE and ATTNRES — but none of those four reaches the log line (crates/dormouse-train/src/lib.rs:1635). So for four of five arms the run log cannot show that the arm ran, and this lane had to infer engagement from the params= header instead. AGENTS.md §2.2 already owes this for attnres; it is now owed for four arms. §1.1: if you add an arm, add its counter — the counters exist, the line does not print them.

The 3-seed control family is retract_every 1; this wave is retract_every 4 (the owner’s working mode). The cadence effect is measured at one seed (6.387 → 6.329, paired at seed 1). Two retract-4 control seeds (2 and 3) would close this at a cost of ~50 min of GPU, and they are the first thing to run when the card is free. Until then every verdict in §6.3 carries a ±0.05-ish systematic on top of the 0.0730 seed range. §6.3 reports both comparisons wherever it matters.

7.3 What this lane did to its own queue (recorded, not hidden)

Section titled “7.3 What this lane did to its own queue (recorded, not hidden)”
  • A mid-flight restart of the driver cost one seed. To lower a disk gate I killed the driver while situ seed 1 was running. The run survived (it lives in its own systemd scope), but the restarted driver launched a second trainer with the same --ckpt-name, whose resume attempt failed, and the refusal branch’s pkill -x train then killed the running seed 1. That is §1.5’s exact failure mode — two processes, one ckpt name — committed by the lane that was supposed to be enforcing §1.5. The ledger row situ 1 REFUSED_AT_RESOLVE in ~/logs/abwave_runs.tsv is that artifact, not an arm refusal.
  • Launch detection was wrong twice before it was right, and both wrong versions looked exactly like a dead arm: systemd-run --user --scope does not wait, so the driver’s first three launches each started a run that nothing was watching. The params=-within-the-first-minute check is what turned a silent failure into a visible one, in 32 seconds.
  • pgrep -f "target/release/train" matches the agent’s own shell, which made tools/first_run.sh refuse with “a train process is already running” while zero train processes existed.
  • No arm here is claimed to be a quality result beyond its own table. One arm (mHC) has a verdict; the rest are partial at the time of writing, and a partial set is not a verdict.
  • Step times in this wave are not a measurement and are not reported as one: every run in §6.3 executed beside other lanes’ CPU work (the DEGRADED LAUNCH rows in ~/logs/abwave_runs.tsv say which, and when). Held-out BPB is a function of the trajectory, not of wall-clock; step time is not, so §3’s 463-515 ms figures are the only step-time numbers in this file and they come from the 100k run, which had the card to itself.
  • The generation comparison is n=1 per cell (§5.3) and its export is step 75 000, not the finished run.