The A/B seed wave — six arms × 3 seeds × 2k steps, and the three measurements that came with it
Lane: GPU queue dispatcher after the 100k run. Started 2026-10-01 13:30, on
main at e638da7. Brief: serialize every CUDA consumer after the 100k
production run, then land the seed wave of the six new arms against the night’s
control family.
Every number here is BEST HELD-OUT bpb from a first_run_2000* log in
~/logs/, over the 81 920 B fixed eval window (20 batches × batch 8 × seq 512
= the window the whole control family shares, §2.6). Two runs with different
batch do not share a window; every run in this file is batch 8.
0. THE CONTROL FAMILY, EXACTLY — and two corrections to the brief
Section titled “0. THE CONTROL FAMILY, EXACTLY — and two corrections to the brief”The brief gave “the control finals were 6.314/6.329/6.387/6.396/6.437/6.443” (6 values), “control mean ≈ 6.37, spread ≈ 0.06”, and seeds 1001/1002/1003. All three need fixing, and the fixes change what the wave means.
0.1 There are SEVEN night logs, and they are TWO families, not one control
Section titled “0.1 There are SEVEN night logs, and they are TWO families, not one control”| log | ckpt name | aux | retract_every | seed | BEST HELD-OUT bpb |
|---|---|---|---|---|---|
first_run_2000_0930_1855.log |
first_run_2000 |
ON (jepa 0.05) | 1 (default) | 1 | 6.387 |
first_run_2000_0930_1921.log |
...--jepa-weight-0---dspark-weight-0- |
OFF (pure CE) | 1 | 1 | 6.437 |
first_run_2000_0930_1940.log |
...--retract-every-4- |
ON | 4 | 1 | 6.329 |
first_run_2000_1001_0037.log |
...--seed-2- |
ON | 1 | 2 | 6.314 |
first_run_2000_1001_0057.log |
...--seed-3- |
ON | 1 | 3 | 6.329 |
first_run_2000_1001_0121.log |
...--seed-2---jepa-weight-0… |
OFF | 1 | 2 | 6.443 |
first_run_2000_1001_0147.log |
...--seed-3---jepa-weight-0… |
OFF | 1 | 3 | 6.396 |
The bolded three are the aux-ON control family. The other four are the A0
(pure-CE) family, already judged in benches/history.tsv (control-3seed
6.387/6.314/6.329 mean 6.343 vs a0-pure-CE-3seed 6.437/6.443/6.396 mean
6.425, “aux arms EARN their place, 3/3 seeds”).
The brief’s 6.37 / 0.06 is the mean and sample-sd of all seven pooled, i.e. of two different models’ recipes averaged together. Pooled: mean 6.3764, sample sd 0.0532, range 0.1290. That is a number about the union of two families and it is the wrong yardstick for a wave that runs aux ON.
The yardstick for this wave is the aux-ON 3-seed control:
| n | mean | sd (sample) | range (max−min) | min | max | |
|---|---|---|---|---|---|---|
| control, aux ON | 3 | 6.3433 | 0.0386 | 0.0730 | 6.314 | 6.387 |
§1.2’s bar is 0.0730 BPB — the control’s own seed range — and a win has to beat it, not beat its own sd. Both numbers are in the table so a reader can apply either reading.
0.2 The seeds are 1/2/3, and “1001/1002/1003” is a filename
Section titled “0.2 The seeds are 1/2/3, and “1001/1002/1003” is a filename”first_run.sh’s log pattern is first_run_${STEPS}_$(date +%m%d_%H%M).log.
first_run_2000_1001_0037.log is Oct 01, 00:37 — 1001 is the date and
0037 the time, not a seed. The seeds on record are 1, 2, 3 (from the ckpt
names --seed-2-, --seed-3- and the seed = 1 in the config snapshot).
So the wave runs seeds 1/2/3, paired against the control’s seeds 1/2/3. This
is deliberate and it is the better design: --seed is a pure function of model
init and the JEPA mask (§3.7), so a same-seed pair starts from the same
initialisation to within the measured TSCT residue (≤1.04e-06 relative), which
removes the init term from the comparison instead of adding a second variance
source. Running seeds 1001/1002/1003 would have compared against a control
measured on 1/2/3 and charged the difference to the arm.
0.3 The recipe, read off checkpoints/first_run_2000--retract-every-4-.config.toml
Section titled “0.3 The recipe, read off checkpoints/first_run_2000--retract-every-4-.config.toml”--preset small --batch 8 --seq-len 512 --no-engram --retract-every 4 --steps 2000 --eval-every 500 --guard --detach --timers --seed <1|2|3>, aux at
the preset defaults (jepa 0.05 + KoLeo, dspark 0.0), eval_batches 20,
lr 1e-4 cosine, opt mix (Muon+ ColRow ns=8), quant auto → Fp8 on
sm_120, max_iter 4, rank 64, n_experts 3, d_model 768, d_ffn 2048,
params 9 197 454. Launched by tools/first_run.sh — the control family was
launched by that same script, and the ckpt-name formula reproduces its names
exactly (--retract-every 4 → first_run_2000--retract-every-4-, which is how
the trailing - on every one of those names is accounted for: tr -c maps the
newline).
A confound I am naming rather than hiding: the 3-seed control family was run
at retract_every 1, and this wave runs retract_every 4 (the owner’s working
mode, and the mode the 100k is on). The cadence effect is measured at one
seed only: seed 1 gives 6.387 (retract 1) vs 6.329 (retract 4), a paired
−0.058. So the arm family is compared against a 3-seed retract-1 control plus a
single retract-4 point. If the wave finishes with time in hand, two retract-4
control seeds (2/3) close this; if it does not, §7 lists it as the open
precision gap.
1. The queue as briefed, and the two places I departed from it
Section titled “1. The queue as briefed, and the two places I departed from it”Brief order: (1) wait for the 100k + record + board, (2) the two-chunk adjoint
CUDA gate in wt/kda-adjoint, (3) generation from the 100k export, (4) the
6-arm × 3-seed wave, (5) per-arm verdict vs the control spread.
Two changes, both forced and both recorded here rather than silently taken:
- Item 2 runs from
main, notwt/kda-adjoint. The brief says the lane’s worktree. Butwt/kda-adjointhas notarget/, and a cold build of one test binary in it is the 686-package / ~41 GB / ~23 min figure the build lock was written for — against 29 GB free on/homeat 94 %. It does not fit, and starting it risks a disk-full mid-link (§2.5: “a mold error that says ‘Disk full?’ is the disk”). Instead:diff -rqofvendor/dormouse-fused/crates/burn-gdn2betweenmainandwt/kda-adjointreports no differences at all except two__pycache__.pycfiles. The lane’s test landed onmainas439d8c8(the same change as itsc3f5c31), andmainhas a warm 98 GBtarget/debug. So the identical gate runs frommainfor the price of one target’s link. - Item 2 is placed after item 3. Item 2 costs a build (~5-15 min) that item 3 does not, and item 4 is the owner’s gate #1 (“all arms A/B’d at 3 seeds”) and needs ~6.3 h of a window that has ~6.5 h left when the 100k exits. Spending 40 min of GPU-serial time on a diagnostic before the deliverable would cost one arm. Item 2 therefore runs immediately after the first arm’s first seed (its build overlaps nothing, and it is bounded at ~2 min of CUDA).
Everything else is as briefed: one CUDA process at a time, pgrep preflight
before every launch, all builds through tools/build_lock.sh run abwave --,
logs in ~/logs/, sequential arms, no push, no deletions, no config-number
changes.
2. The six arms, and what each one’s flag is
Section titled “2. The six arms, and what each one’s flag is”Flags read from the arms’ own landing reports and docs/protocols/AB-PROTOCOL.md, not
guessed. Every arm is off by default for checkpoint compatibility, and each is
a single --set into the resolved config (--set is a generic serde override,
config/override.rs:11, so every schema field is reachable without a typed
flag).
| # | arm | flag(s) | source | param delta on small |
what proves it engaged |
|---|---|---|---|---|---|
| 1 | mHC (manifold-constrained hyper-connections) | --set use_mhc=true |
mhc-2026-09-30.md (row 7b), schema.rs:123 |
+0.067 % at the default mhc_streams = 2 |
params= on the log header line moves; probe::MHC exists but is not printed |
| 2 | SiTU-GLU (Kimi K3 Eq 12) | --set use_situ=true |
situ-2026-09-30.md, AB row 9, schema.rs:140 |
+393 216 (+4.28 %) | params=9197454 → 9590670 in the header |
| 3 | RoPE on the KDA q/k | --set use_kda_rope=true |
AB row 8 | — | see §2.1 — the field does not exist on main |
| 4 | MoE routing | --set moe_topk=1 |
moe-routing-2026-10-01.md, schema.rs (d_moe_topk() = 0) |
0 (param-neutral by construction) | the resolved-config snapshot + moe_lb_coef left at 0.0; see §2.2 |
| 5 | AttnRes | --set use_attnres=true |
AB row 7, schema.rs:122 |
the arm’s d_model-wide query vectors |
params= moves; probe::ATTNRES exists but is not printed (already owed, AGENTS §2.2) |
| 6 | future-byte head | --set aux_fb_weight=0.1 |
AB row 1b, future-byte-2026-09-30.md |
+197 120 (+2.1 %) | params=9197454 → 9394574 and the log line’s fb=<ran>/<asked>, the only arm counter the line actually prints |
2.1 RoPE — the arm is not reachable from main, and that is a fact, not a bug to debug here
Section titled “2.1 RoPE — the arm is not reachable from main, and that is a fact, not a bug to debug here”rope-2026-09-30.md:275-277 says so in its own words: “Not wired to the
trainer’s config or CLI. The flag is on KdaConfig … wiring it into
dormouse-core’s DormouseConfig and a CLI flag is the owner’s call about
§5.” grep -i rope over schema.rs and cfg.rs on main returns nothing
model-shaped. So --set use_kda_rope=true on main is an unknown config
key, which config/override.rs refuses loudly. Per the brief’s own rule
(“if an arm refuses to run, record the refusal LOUDLY and skip”) this is
recorded as a refusal, and no code is touched: wiring a flag is a code change
to another lane’s crate, not a dispatcher action. It is also the one row the
protocol says to run as a post-training gate, not a BPB race (AB row 8), so
it was never going to be decided by this wave.
2.2 MoE — top-1 of 3, and why not 4
Section titled “2.2 MoE — top-1 of 3, and why not 4”The arm’s own fixture and its headline number are 4 experts / top-1
(moe-routing-2026-10-01.md:1, “dense shared FFN vs 4× top-1 at equal ACTIVE
FFN FLOPs”), and small ships n_experts = 3. Raising n_experts to 4 to
match would add a fourth expert’s worth of parameters and turn a
param-neutral routing A/B into a capacity A/B, so the wave runs the shipped
n_experts = 3 with --set moe_topk=1: top-1 of 3, still param-neutral
(num_params equal at moe_topk 0 and 1 is the crate’s own gate), which is the
property that makes it comparable to the control at all. moe_lb_coef stays
0.0 — the report ships no coefficient (“no coefficient is shipped, and
moe_lb_coef stays 0.0 with the arm off”), and lb > 0 with routing off is
refused.
Stated limit on this arm’s evidence: routing is the one arm here whose
on-position does not change params=, and probe::MOE_ROUTE is not on the
log line (only fb= and the legacy seams are). So for MoE, “the arm ran” is
carried by the resolved-config snapshot plus the crate’s own
off_is_the_dense_blend_bit_for_bit gate, not by a runtime counter. A silently
un-engaged MoE run would be indistinguishable from the control in the log. That
is a §1.1 (SILENT) gap in the log line, not in the model — listed in §7.
3. The 100k production run — final
Section titled “3. The 100k production run — final”It finished at 14:42 on its own, with nothing done to it. Final line, verbatim:
done steps=100000 best ce=2.415 | BEST HELD-OUT bpb=5.545 at step 86000 -> checkpoints/first_run_100000--retract-every-4-.best.bin| steps | 100 000, complete (planned 100 000) |
| wall clock | 2026-10-01 02:02 → 14:42 = 12 h 40 m, ~456 ms/step averaged; warm steps read 463-515 ms |
| BEST HELD-OUT bpb | 5.5452847 at step 86 000 (sidecar ….best.bpb, agrees with the log line) |
| final eval | 5.729 at step 99 500 |
| best train ce | 2.415 |
| NaN / firewall / panic / guard-restart | 0 lines of all four in 2 126 log lines |
| window | 81 920 B fixed, 200 evals |
| commit | the release binary was built 09-30 18:31, off a7c3cad — which is before all six arm landings, so this run provably contains none of them |
| board | python3 tools/trainboard.py --out /tmp/opencode/board-final ~/logs/first_run_100000_1001_0202.log → 4 PNGs + index, regenerated 14:44 |
| history | one row appended to benches/history.tsv (2026-10-01T11:43:00Z, commit a7c3cad) |
Reading it honestly. The curve descends 6.480 @ 500 → 6.042 @ 40 500 → 5.769 @ 50 500 → 5.720 @ 75 500 → 5.603 @ 85 500 → 5.545 @ 86 000, and then it does not descend again: 5.595, 5.603, 5.666, 5.692, 5.740 for the last 14 k steps. So the number is best-of-a-plateau, not a late win, and the last 14 k steps (11 % of the run, 1 h 45 m) bought nothing — that is the same shape as §3.1’s 4.997 row, and the same honest reading applies. Against the 2k controls (6.343 aux-ON / 6.425 pure-CE, §0.1) the 100k run is 0.80 BPB better, so the budget itself is worth ~0.8 BPB over 50× the steps.
The board’s second trace makes the regime visible: at step 100 000 the train
BPB is ~3.7 and the held-out BPB is 5.73. A 2.0-BPB train/held-out gap on a
held-out set of 81 920 B is §2.3’s overfit class, not a measurement artefact —
which is also why the last 14 k steps are flat: the model is spending them
fitting bytes it has already seen. The next budget question is therefore not
“more steps” (the curve says no) but “more distinct bytes” (queue row 3,
working set) or a larger model (the owner’s ×10 gate, .bulba/goal.md).
Two counter readings from the same log, because they are on the line and a reader will trip over them:
fused kda=398012/0 bwd=0 … declined=2037796 ops=811936 node_bwd=0. The fused forward ran 398 012 times and its result is discarded; the gradient path was the ops arm every step. That is §3.3’s FINDING 2 (ADR-0019’s silent shape: a run that launches a kernel and throws the answer away), and this run is another instance of it, not a new defect.norm=0/911936: the fused RMSNorm kernel was asked 911 936 times and ran zero — the reachability story of §3.3, unchanged, on a 100k-step run.
4. The two-chunk fused-adjoint gate on CUDA — GREEN, and deterministic
Section titled “4. The two-chunk fused-adjoint gate on CUDA — GREEN, and deterministic”Run 2026-10-01 15:21-15:27, card otherwise idle (checked immediately before each
launch), test result: ok. 1 passed; 0 failed, 3.2-5.7 s per run.
How it was run, and why not in wt/kda-adjoint. The brief said to run it
there; §1 explains why that was not possible (no target/, 41 GB needed, 23 GB
free on /home). What ran instead was the same test binary the lane had
already built:
/mnt/e43497ab-0ff2-45b4-b45f-28de3339a53e/kdagf-target/debug/deps/fused_adjoint_f64-26e48d2e1534219bLD_LIBRARY_PATH=<main>/target/debug/build/openblas-src-*/out/OpenBLAS-0.3.32 --nocapture --test-threads=1Three provenance facts make that the same measurement, not a shortcut around it:
diff -rq of vendor/dormouse-fused/crates/burn-gdn2 between main and
wt/kda-adjoint reports no differences (only two __pycache__ .pyc); the
test file’s mtime in that worktree is 11:38:53, the binary’s is 12:07:49,
so the binary post-dates the two-chunk edit; and the run prints which scope it
is in, so the binary proves its own vintage — both === ONE CHUNK (BK1 only) ===
and === TWO CHUNKS (BK1 + BK2 + the d_s_shift glue) === appear. The lane’s
LD_LIBRARY_PATH was needed because openblas-src emits its .so into the
build dir with no rpath; the file lives in this worktree’s target, which is
where the lane got it from too.
4.1 The numbers (bar: 1e-3 relative on the f32 side; oracle self-consistency ~1e-12)
Section titled “4.1 The numbers (bar: 1e-3 relative on the f32 side; oracle self-consistency ~1e-12)”| input | ONE CHUNK (BK1) | TWO CHUNKS (BK1 + BK2 + d_s_shift) |
|---|---|---|
| fused forward vs the f64 transcription | rel 2.381e-7 | rel 2.094e-7 |
| d_q | 2.416e-7 | 2.605e-7 |
| d_k | 1.642e-7 | 1.771e-7 |
| d_v | 2.419e-7 | 3.713e-7 |
| d_g | 2.496e-7 | 4.410e-7 |
| d_b | 1.402e-7 | 2.044e-7 |
| d_w | 2.329e-7 | 2.248e-7 |
| d_state | 2.655e-7 | 1.782e-7 |
Every one of the fourteen cells is ok, and the two-chunk arm is 1.4-2.5×
tighter than its bar, not near it.
4.2 Six runs, because “green once” cannot decide a race
Section titled “4.2 Six runs, because “green once” cannot decide a race”The gate costs 3-6 s, so it was run six times. All six: 1 passed; 0 failed,
and all seven two-chunk maxima identical to 7 significant figures
(q=2.605e-7 k=1.771e-7 v=3.713e-7 g=4.410e-7 b=2.044e-7 w=2.248e-7 state=1.782e-7), as were all seven one-chunk maxima.
4.3 The verdict, and the three things it is not
Section titled “4.3 The verdict, and the three things it is not”GREEN and reproducible at the two-chunk scope. BK2 — the sequential reverse
recurrence over chunks, plus the d_k_bptt / d_e_bptt / d_s_shift glue that
2a430cc measured at rel 3.3e-1 and 6.2e-1 before its fix — matches an
independent f64 oracle to 4.4e-7 at worst, and does not move across runs.
So on the reading docs/reviews/kda-adjoint-2026-10-01.md §4 offers as (a) —
f32 conditioning of the instrument, not a race in the kernel — the evidence
lands on (a): a term that is ~1e-4 of |L| is exactly the magnitude this oracle
resolves to 4e-7, so an f32 finite difference of it moving 7-27 % is the
instrument’s error budget, not the kernel’s.
Stated limits, so nobody reads this as closing more than it closes:
- Scope is 2 chunks (T=32), and
d8fa449’s disagreement is at 16 chunks (T=256). Different scope, and the 16-chunk disagreement is not resolved by this run — it is only shown to be a scope question, not a broken BK2. - The instrument is an f64 oracle, not a finite difference. So this run does not speak to the precision of the FD instrument itself; it speaks to the kernel.
- Determinism is measured at the seven printed maxima, i.e. to 7 significant figures of an aggregate. Identical aggregates are strong evidence and not a bitwise proof of every output element.
- BK2-gated does not mean BK2-enabled. §6 of that review is unchanged: on the trainer’s backend the fused op declines, so the ops path still carries the gradient, and nothing here argues for turning the kernel on.
(§4’s verdict is also the answer to the brief’s premise that this run “does not exist”: a run of exactly this binary was on the card at 14:43, ~20 min before I re-ran it myself, and my numbers are independent of whatever that one printed.)
5. Generation from the 100k export vs the 2k export
Section titled “5. Generation from the 100k export vs the 2k export”Run 2026-10-01 15:32-15:40 with ./target/release/generate, the binary rebuilt
from main at 15:19.
5.1 The first thing to say: it is not the 100k model
Section titled “5.1 The first thing to say: it is not the 100k model”--export checkpoints/first_run_100000--retract-every-4-.f32.dmexp prints
loaded … (F32 weights, 9197454 params, **step 75000**). The export was written
at 11:20 from a step-75 000 checkpoint, so every sample below is a step-75 000
model, not the finished run (whose best held-out point is 5.545 at step 86 000,
§3). The 2k export is likewise step 1500, its own best point. So the comparison
is 75k-steps-vs-1.5k-steps of held-out BPB 5.72 vs 6.39, which is the honest
framing of every row below.
5.2 --temp 0 does not exist: it panics
Section titled “5.2 --temp 0 does not exist: it panics”$ ./target/release/generate --export … --prompt The --steps 120 --temp 0cannot sample empty rangeThe sampler computes exp((x - max) / temp); at temp = 0 the winning element is
0/0 = NaN, the sum is NaN, and gen_range refuses the range. So the brief’s
“temperature 0 (greedy)” cell is not available in this binary, and the honest
thing is to record the panic rather than substitute a near-zero temperature and
call it greedy. Greedy decode is a one-line branch (argmax when temp <= 0) in
crates/dormouse-cli/src/bin/generate.rs:47; it is the inference owner’s call, not
this lane’s, and it is listed in §7.
5.3 What the models actually emit, counted
Section titled “5.3 What the models actually emit, counted”Prompt The, 150-300 steps, same binary, same export format. “real” = a token in
a 100-word common-English list; digits/upper/punct = how much of the output
is symbol soup.
| temp | model | bytes | tokens | real words | distinct | digits | uppercase | punct |
|---|---|---|---|---|---|---|---|---|
| 0.15 | 100k, step 75 000 | 153 | 32 | 12 — 37.5 % | 24 | 0 | 1 | 0 |
| 0.15 | 2k, step 1 500 | 146 | 25 | 4 — 16.0 % | 24 | 1 | 2 | 2 |
| 0.30 | 100k, step 75 000 | 147 | 15 | 2 — 13.3 % | 15 | 10 | 6 | 6 |
| 0.30 | 2k, step 1 500 | 152 | 22 | 2 — 9.1 % | 22 | 15 | 15 | 15 |
| 0.80 | 100k, step 75 000 | 221 | 43 | 5 — 11.6 % | 37 | 39 | 47 | 17 |
| 0.80 | 2k, step 1 500 | 283 | 56 | 1 — 1.8 % | 44 | 33 | 70 | 29 |
At temp 0.15, verbatim:
100k: Thers o the woreronated the the thicecor t the to thartonds tor the the isthe cons on to tho the alor intritow bo toler s or cof cutiron fero bu the thet2k : Themanseve th us s anteg sengonatorsintwe s te Lis en tere melemisons benegit f es wige ( meatiollinsgelere of ty on as ged m1 \( imiorosome bethoThe 100k sample contains 8 occurrences of “the” and no digits and no
uppercase; the 2k sample contains zero and drifts into (, \ and
symbol soup. So the trained model has real English function-word structure where
the 2k model has none.
Verdict, honestly bounded:
- At the archive’s temperature (0.8) the comparison says nothing about either model. Both are noise; a model at 2.3 bits/byte has no readable 0.8-temperature sample to give. The 100k number is nominally better (11.6 % vs 1.8 %) and that ordering does survive at 0.15 and 0.30 — but the absolute quality is “fragments”.
- The brief’s baseline is not reproducible as described. I could not get the 2k model to emit anything like “The the the in con ate” at any of the three temperatures; at 0.8 its output has no repeated “the” at all. Either that sample came from a different temperature, a different build, or a different prompt. I am not claiming to have reproduced it, and the comparison above is against a freshly generated 2k sample, not against that string.
- n = 1 sample per cell. These are counts on one 150-byte continuation each. They rank the two models consistently across three temperatures; they are not a statistic and must not be quoted as one.
- A byte-level 9.2M model at step 75 000 is not a text model yet. The honest summary is that 100k steps bought function-word structure that 2k steps did not have — visible in the counts, invisible in anything you would want to read.
6. The wave — per-arm table
Section titled “6. The wave — per-arm table”6.1 The finding that reshaped the queue: four of the six arms could not be set
Section titled “6.1 The finding that reshaped the queue: four of the six arms could not be set”The first launch attempt was --set use_mhc=true, the invocation
docs/reviews/mhc-2026-09-30.md and AB row 7b document. It died in 32
seconds:
train failed: unknown config key "use_mhc"--set is not a serde override. It is a hand-written match
(crates/dormouse-core/src/config/override.rs:28-71) listing 33 keys, and
use_mhc, mhc_streams, use_situ and use_attnres are not on it. The
config fields exist in schema.rs and the arms are wired into the model; the
trainer’s seam that turns a command-line string into a config value was never
extended for them. So the four documented invocations
(--set use_mhc=true, --set use_situ=true, --set use_attnres=true,
--set use_kda_rope=true) have never reached a trainer process. Two of the
six arms’ flags are on the list — moe_topk, aux_fb_weight — which is why
this went unnoticed for a week: the two arms that were documented as runnable
were the two that worked.
This is the ADR-0011 LOUD class doing its job (the refusal named the cause and
the escape), and §1.7’s document-vs-code disagreement class. The fix is six lines
in override.rs; it is not this lane’s to make (§1.6: do not edit files
another lane is in), so it is a follow-up in §7 with the file:line.
The reachable seam, used instead: loader.rs’s is_explicit() accepts any
string containing / or ending .toml as an explicit path, and
DormouseConfig deserializes from TOML with #[serde(default)] on every field.
So /tmp/opencode/presets/<arm>.toml — configs/small.toml verbatim plus one
arm line, verified by diff — reaches all five arms with no code change:
$ diff <(grep -v '^#' /tmp/opencode/presets/mhc.toml) <(grep -v '^#' configs/small.toml)31,32d30< use_mhc = true< mhc_streams = 2Every arm’s preset is configs/small.toml plus one or two lines and nothing
else changed, and no preset file was added to configs/ (the owner’s directory).
6.2 Arm status
Section titled “6.2 Arm status”| arm | flag as documented | reachable? | how it was run |
|---|---|---|---|
| mHC | --set use_mhc=true |
no (not in override.rs) |
presets/mhc.toml: use_mhc = true, mhc_streams = 2 |
| SiTU | --set use_situ=true |
no | presets/situ.toml: use_situ = true |
| AttnRes | --set use_attnres=true |
no | presets/attnres.toml: use_attnres = true |
| MoE | --set moe_topk=1 |
yes | presets/moe.toml: moe_topk = 1 (moe_lb_coef stays 0.0) |
| future-byte | --set aux_fb_weight=0.1 |
yes | presets/fb.toml: aux_fb_weight = 0.1 (horizon 2) |
| RoPE | --set use_kda_rope=true |
no field at all | REFUSED — no schema field, no preset, nothing to run |
6.3 Results
Section titled “6.3 Results”(per-arm rows land here as runs complete; the yardstick is §0.1: control aux-ON mean 6.3433, §1.2 bar = the control’s own seed range 0.0730)
Every run below: --preset = configs/small.toml + the arm line(s), batch 8,
seq 512, --retract-every 4, 2000 steps, seeds 1/2/3, aux jepa 0.05 + KoLeo,
window 81 920 B, 0 NaN, 0 panics, BEST HELD-OUT bpb from the log’s own last line.
6.4 The arms, one by one
Section titled “6.4 The arms, one by one”mHC (use_mhc = true, mhc_streams = 2) — TIE
Section titled “mHC (use_mhc = true, mhc_streams = 2) — TIE”| seed | arm | control (same seed) | paired Δ |
|---|---|---|---|
| 1 | 6.4130 | 6.387 | +0.026 |
| 2 | 6.3130 | 6.314 | −0.001 |
| 3 | 6.3700 | 6.329 | +0.041 |
| mean | 6.3653 (sd 0.0502, range 0.100) | 6.3433 (sd 0.0386, range 0.0730) | +0.0220 |
Verdict: TIE. |Δ| = 0.0220 against a bar of 0.0730 — the arm is inside the control’s own seed spread, and §1.2 says a tie deletes the mechanism. Two things make this verdict stronger than a bare number:
- The arm engaged, so the tie is a measurement and not a no-op:
params=9203609in every log against the control’s9197454, i.e. +6155 params = +0.067 %, which is exactly the costmhc-2026-09-30.mdquotes for n=2. Same number, independently. - The report predicted this outcome from the mechanism, not from the curve:
at the φ-lever’s left init the composite sits at the identity, so the arm
starts from the same function as ReZero and differs in the operator — and the
owner’s own note on it already reads “пока не внедрять в дефолт”. This wave
turns that expectation into a 3-seed tie.
mhc_streams = 4(the base paper’s own rung) remains the follow-up row the report names, and this result says nothing about it.
Against the exact-recipe control (the one paired --retract-every 4 seed-1
point, 6.329) the delta is +0.084, i.e. outside the bar in the losing direction —
but that is a single seed and cannot carry a verdict, which is why §0.3
names the two missing retract-4 control seeds as this wave’s precision gap.
AttnRes and future-byte — run by the wave3 lane, NOT by this one
Section titled “AttnRes and future-byte — run by the wave3 lane, NOT by this one”This dispatcher’s queue died before these two landed; the runs are in
benches/history.tsv at 2026-10-02T04:25:00Z (commit af9def1) and are
summarised here rather than re-run, because re-running them would spend card
time to reproduce numbers that are already on disk with the same window, the same
bar and the same verdict rule. Full rows: wave3-attnres-3seed-FRESH,
wave3-fb-3seed-FRESH.
| arm | seeds | mean | vs control 6.3433 | verdict |
|---|---|---|---|---|
AttnRes (use_attnres) |
6.191 / 6.346 / 6.453 | 6.3300 | −0.013 | TIE → deleted (§1.2); its own seed range is 0.262 = 3.6× the control’s 0.073 |
future-byte (aux_fb_weight = 0.1) |
6.406 / 6.391 / 6.463 | 6.4200 | +0.077 | TIE → deleted (§1.2); widest loss of the wave, and its train CE (2.998-3.040) is the worst of any completed arm |
Both arms verified engaged from the log: AttnRes params=9200526 (+3072),
future-byte fb=1503/1503 and params=9394318 (+196 864). The AttnRes seed
spread is itself the finding: a 0.262 range on three seeds of the same recipe
means an AttnRes “win” of 0.05 BPB would have been noise, and it is 3.6× the
control’s spread, so §1.2’s bar is not even the right width for that arm.
SiTU, MoE, RoPE — no result exists
Section titled “SiTU, MoE, RoPE — no result exists”| arm | why there is no number |
|---|---|
| SiTU | all three of this lane’s runs died at step 0. Same signature four times (see §6.5): the process goes single-threaded at ~97 % CPU, makes zero syscalls and zero I/O, and its GPU memory drops from ~9 GB to 699 MiB. It is a spin, not training. One run (attnres seed 2) got to step 200 before dying the same way. The cause was never isolated — the three days since have no core dump, no strace (not installed), and no surviving process to inspect. |
| MoE | this lane’s own fault, and the mechanism is worth recording: my driver killed all three runs at 45 s with “ARM DID NOT ENGAGE”, because it compared params= against the control’s count. MoE is param-neutral by construction (num_params equal at moe_topk 0 and 1 is the crate’s own gate), so a param check cannot detect it — I killed three correct runs with a check that was only ever valid for arms that add parameters. The MoE lane later shipped tools/moe_ab.sh with its own preflight and recorded “ready, NOT run: no GPU, night2e owned the card all lane, so NO run and NO A/B result” (benches/history.tsv, 2026-10-03). MoE still has no A/B number. |
| RoPE | no schema field. --set use_kda_rope=true has nothing to set, and docs/reviews/rope-2026-09-30.md:275-277 says wiring it to the trainer’s config is “the owner’s call”. Not run, not runnable. |
6.5 The step-0 death signature (four runs, unexplained)
Section titled “6.5 The step-0 death signature (four runs, unexplained)”The four runs that died at step 0 (situ seeds 1/2/3, attnres seed 1) all
left a log of exactly 8 lines ending ckpt … saved step 0, and a process in
this state:
elapsed 31:25 cpu 00:30:33 97.2 %CPU 1 thread RSS 5.8 GB State: R/proc/PID/io: rchar and wchar FROZEN over a 10 s sample (zero I/O)/proc/PID/syscall: 0 (no syscall in flight)nvidia-smi: utilisation 6 %, 699 MiB used (a training step uses ~9 GB)One thread, no syscalls, no I/O, GPU memory released: a userspace spin that has
already given the device back. It is not an OOM (the ram-guard log’s last
entry is 2026-09-30, and MemoryMax=40G never tripped), not a NaN firewall
(the log would say so), and not a disk-full kill (that signature appears in
the archive as ENOSPC, from the 02:05 event of 2026-10-02, and the disk had
70 GB free when these died). It is an open defect, not a result. The cheapest
next step is one run under --memlog with DM_QUANT_DEBUG=1, or a core dump on
the spin, and it belongs to whoever owns the trainer’s step boundary.
6.6 The wave’s final tally
Section titled “6.6 The wave’s final tally”| # | arm | seeds | mean held-out BPB | Δ vs control 6.3433 | §1.2 verdict |
|---|---|---|---|---|---|
| 1 | mHC | 3 (this lane) | 6.3653 | +0.022 | TIE → deleted |
| 2 | SiTU | 3 attempted, 0 completed | — | — | no result (step-0 spin, §6.5) |
| 3 | RoPE | not runnable | — | — | no result (no schema field) |
| 4 | MoE | 3 attempted, 0 completed | — | — | no result (killed by my own check; the MoE lane’s own run also never happened) |
| 5 | AttnRes | 3 (wave3) |
6.3300 | −0.013 | TIE → deleted |
| 6 | future-byte | 3 (wave3) |
6.4200 | +0.077 | TIE → deleted |
Three of six arms are judged, and all three are ties, which under §1.2 means all three are deleted. None of the three that reached a number earned its place: the best of them (AttnRes, −0.013) is a sixth of the control’s own seed range, and the two with the largest sample spread are the two whose “signal” would have been most likely to be noise. Three arms remain unmeasured, and one of those (MoE) is the arm the queue’s own §3.5 playbook cares most about at 1B.
That is the honest shape of this wave: it did not find a winner, and the
mechanism it was built to find is now three deletions and three open questions.
The cheapest next card-time is SiTU and MoE at 3 seeds each (~2.5 h), which is
also the only way the “arms hurt at 9.2M” reading in
benches/history.tsv:4c7721f (byteflow 3.466 / plain9m 3.947 / dormouse
6.498) gets tested on the arms rather than on the whole architecture.
7. What is NOT claimed, and the follow-ups this lane owes
Section titled “7. What is NOT claimed, and the follow-ups this lane owes”7.1 Three defects this lane found, all in the seam the A/B queue runs through
Section titled “7.1 Three defects this lane found, all in the seam the A/B queue runs through”--setcould not reach four of the six arms (§6.1) — FIXED, 2026-10-01 21:24,68cb247“fix(core): –set knows the four queue arms it loudly refused”, 2 h 40 min after this lane reported it. At the time of writing this sectionoverride.rswas a hand-writtenmatchover 33 keys withuse_mhc,mhc_streams,use_situanduse_attnresabsent from it, so--set use_mhc=true,--set use_situ=trueand--set use_attnres=truehad never reached a trainer; the documented invocations are live now. This wave therefore ran its arms through preset files, which is why its own logs name/tmp/opencode/presets/*.tomlrather than a flag — that path is still the right one for any arm that adds a field the seam does not know.generatehas no greedy decode (§5.2):--temp 0divides by zero, produces NaN, and dies withcannot sample empty range.crates/dormouse-cli/src/bin/generate.rs:47—argmaxwhentemp <= 0is the whole fix.- The log line prints one arm’s counter.
fb=<ran>/<asked>is there, andprobe.rscountsMHC,SITU,MOE_ROUTEandATTNRES— but none of those four reaches the log line (crates/dormouse-train/src/lib.rs:1635). So for four of five arms the run log cannot show that the arm ran, and this lane had to infer engagement from theparams=header instead. AGENTS.md §2.2 already owes this forattnres; it is now owed for four arms. §1.1: if you add an arm, add its counter — the counters exist, the line does not print them.
7.2 The retract_every confound, unclosed
Section titled “7.2 The retract_every confound, unclosed”The 3-seed control family is retract_every 1; this wave is retract_every 4
(the owner’s working mode). The cadence effect is measured at one seed
(6.387 → 6.329, paired at seed 1). Two retract-4 control seeds (2 and 3) would
close this at a cost of ~50 min of GPU, and they are the first thing to run when
the card is free. Until then every verdict in §6.3 carries a ±0.05-ish
systematic on top of the 0.0730 seed range. §6.3 reports both comparisons
wherever it matters.
7.3 What this lane did to its own queue (recorded, not hidden)
Section titled “7.3 What this lane did to its own queue (recorded, not hidden)”- A mid-flight restart of the driver cost one seed. To lower a disk gate I
killed the driver while
situseed 1 was running. The run survived (it lives in its own systemd scope), but the restarted driver launched a second trainer with the same--ckpt-name, whose resume attempt failed, and the refusal branch’spkill -x trainthen killed the running seed 1. That is §1.5’s exact failure mode — two processes, one ckpt name — committed by the lane that was supposed to be enforcing §1.5. The ledger rowsitu 1 REFUSED_AT_RESOLVEin~/logs/abwave_runs.tsvis that artifact, not an arm refusal. - Launch detection was wrong twice before it was right, and both wrong
versions looked exactly like a dead arm:
systemd-run --user --scopedoes not wait, so the driver’s first three launches each started a run that nothing was watching. Theparams=-within-the-first-minute check is what turned a silent failure into a visible one, in 32 seconds. pgrep -f "target/release/train"matches the agent’s own shell, which madetools/first_run.shrefuse with “a train process is already running” while zero train processes existed.
7.4 Not claimed
Section titled “7.4 Not claimed”- No arm here is claimed to be a quality result beyond its own table. One arm (mHC) has a verdict; the rest are partial at the time of writing, and a partial set is not a verdict.
- Step times in this wave are not a measurement and are not reported as one:
every run in §6.3 executed beside other lanes’ CPU work (the
DEGRADED LAUNCHrows in~/logs/abwave_runs.tsvsay which, and when). Held-out BPB is a function of the trajectory, not of wall-clock; step time is not, so §3’s 463-515 ms figures are the only step-time numbers in this file and they come from the 100k run, which had the card to itself. - The generation comparison is n=1 per cell (§5.3) and its export is step 75 000, not the finished run.