Night operator findings — 2026-10-02

Lane: night build + verdicts + the overnight run. Owner asleep; everything here carries its command and the log it came from. Companion documents: docs/reviews/night-verdicts-2026-10-02.md (Phase 2 A/B table), docs/reviews/unguard-2026-10-02.md (the guard removal that explains the wave’s history), research/reviews/graph-trainer-2026-10-01.md (the graph gate this night settles, in wt/graph-trainer until it merges), docs/reviews/f16-tensor-core-2026-10-01.md (the f16 gate).

Phase 1 — wave health (01:19-01:35, while the wave runs)

The wave on disk is wave 3 (~/logs/ab_wave3.log, runner reads /tmp/opencode/wave_plain.sh, 15 commands, one CUDA process at a time via a pgrep -x train guard between runs — the guard works because each command’s --detach parent exits 0 and the runner waits out the detached child instead; the unguard lesson is applied in the runner, not in train).

• Wave 2 (22:25-22:26) is carnage and is NOT evidence: mhc_s1/s2/s3, fb_s1/s2/s3, attnres_s1/s2 and situ_s2 all died at startup or within seconds — failed to reserve 1073741824 bytes — during the unguard stack era (fifteen runs on one GPU, docs/reviews/unguard-2026-10-02.md). Their logs were partially overwritten by wave 3; what remains of them is not a measurement.
• Wave 3 so far (STARTs 00:08 → 01:12):
  • situ_s1: dead at step ~300, CUDA OOM mid-run, solo on the card (failed to reserve 67108864 bytes → train failed: step 300: device error (loss is not a scalar)).
  • situ_s2: no wave-3 output in its log (instant death, log mtime still 22:25) — effectively no data.
  • situ_s3: dead at step ~200, same OOM shape (33 MB reserve fails).
  • moe_s1: completed, BEST HELD-OUT 6.359 @ step 1500.
  • moe_s2: completed, BEST HELD-OUT 6.303 @ step 1500.
  • moe_s3: running at 01:21 (~657 ms/step at step 400, 3.6 GB VRAM).
• The situ OOMs happened with one process on the card (the runner’s guard serializes), which makes them an arm defect, not contention: 2/2 seeds that ran died at step 200-300 with the pool failing on 32-67 MB reserves. Verdict implication regardless of the remaining seed: situ cannot ship in the night build. Recorded as a follow-up for the arm’s owner.
• Remaining after moe_s3: attnres s1-3, mhc s1-3, fb s1-3 (9 runs). At the observed ~22-29 min/run (moe_s1 00:20→00:49, moe_s2 00:49→01:11), WAVE_DONE ≈ 05:00-05:45.

02:05 — INCIDENT: another session’s CI build filled /home and killed a wave run

attnres_s2 died at step 1000: checkpoint save failed: No space left on device (os error 28) — /home hit 99 % (6 GB free), after 37 GB grew in ~40 min. Cause found and stopped: a live parallel session (ses_f3b8b1524ffe0…) was running /tmp/opencode/cifinal/heavy-c.sh under the build lock (cifinal2) — a 6-step vendor CI pass building into dormouse-wt/ci-final/…/target (39 GB of artifacts). Its first five steps were green; it was on the last (cargo test --release -p burn-gdn2 --features binary-tests) when terminated.

Actions taken (02:05-02:10):

• SIGTERM to the heavy-c tree (script PIDs 301552/300888/300889, cargo 378910, openblas build-script 387626). A breadcrumb appended to ITS log (/tmp/opencode/cifinal/heavy-c.log) explaining why. Its own one-shot watcher sleeps will report the kill to its session.
• Reclaimed the regenerable build artifacts only: ci-final/target (6.8 GB) + ci-final/vendor/dormouse-fused/target (32 GB). Worktree sources untouched. /home back to 91 % / 46 GB free.
• Casualty: attnres_s2’s wave-3 run (fresh, was at step 1000 of 2000, its step-1000 eval had just printed 6.497). NOT evidence for the verdicts; the arm needs the fresh-seed re-run (below).
• The build lock is free again (cifinal2 holder is gone) — the post-wave gate build no longer queues behind it.
• mhc_s2 (running during the incident) completed fine — it is a no-train resume of wave 1’s checkpoint (see below) and only reads.

Follow-ups for the owner (not fixed tonight): (1) ~80 GB more is reclaimable from landed lanes’ worktree targets (dormouse-wt is 163 GB: dedup-opt 37 G, gdn2bwd 28 G, config-seams 10 G — all merged branches, git worktree list); (2) whatever session owns heavy-c.sh should re-run its last step when the disk budget allows; (3) the checkpoint save path treats ENOSPC as a run-fatal error — correct (loud), but a pre-flight df check like §2.4’s would have caught it before burning 15 min of GPU.

The no-train resumes, precisely (verified 02:00-02:10)

• Wave 2’s attnres runs (s1/s2/s3) all died at step 0 during the OOM storm — their leftovers are step-0 checkpoints (s2’s .txt: step 0 ce 5.574). Wave 3’s s1 and s2 therefore loaded a step-0 checkpoint (no resumed banner — it prints only for step > 0) and ran full fresh 2000-step runs from the seed init: valid fresh evidence.
• attnres_s3 and mhc_s1/s2/s3 resume completed prior-wave checkpoints: stream resume: skipped 8196096 bytes (step 2000 …), zero training, and the “BEST HELD-OUT” line is the prior wave’s number re-printed (attnres_s3 6.406 = wave 2’s own best.bpb; mhc 6.413/6.313 = wave 1’s published TIE rows). Also note best ce=inf in those lines — the CE best is not carried in the .bpb restore path. These rows are counted as their prior waves’ evidence, not new measurements; the verdict table marks them.

Two earlier waves left artifacts under the SAME ckpt-names wave 3 reuses (ab2000_{arm}_s{n}): wave 1’s completed mhc runs (17:12-18:10, the source of the already-recorded mHC TIE, 66a8f0d) and wave 2’s partial/complete attnres runs (21:44-23:40). The trainer resumes by ckpt_name (the .bin/.prev.bin candidates, corrupt ones refused loudly at load with a finite-scan), guarded by the ADR-0021 snapshot diff (hard error on any non-progress key; source is provenance and excluded by design, cfg.rs:100-140).

01:35-01:55 — the prior-wave leftover question, settled (first pass)

Verified per run, from each log’s LAST segment (after the final dormouse pretrain header — the logs APPEND across waves):

• moe s1/s2/s3 (wave 3): fresh, step-0 ce 5.571/5.576/5.576, both leftover candidates refused (“unusable” ×2 each), full 2000 steps.
• attnres_s1 (wave 3): fresh — its own header at line 115, step 0 ce=5.575, ckpt ... saved step 0, curve identical in shape to moe’s fresh seed-1 run (3.296@200 / 3.153@300 — determinism making two independent seed-1 runs nearly indistinguishable, which is the point of --seed). The earlier suspicion of a silent resume was wrong; a real resume prints resumed {name} from {dir} step {N} (lib.rs:927) and none printed.
• Predicted for the rest of the wave: attnres_s2 (wave-2 partial .bin at 22:20 on disk) and attnres_s3 / mhc_s1-3 (wave-1/2 completed checkpoints) may legitimately RESUME — s2 continues a partial run (valid), but a completed checkpoint resumes at step 2000, trains nothing, and its “BEST HELD-OUT” line is the PRIOR wave’s number (attnres_s3’s was measured on a card where sibling processes were OOM-crashing — the stacking era). The extractor classifies each segment (/tmp/opencode/verdicts.sh); NO-TRAIN-RESUME segments are not wave-3 evidence and are reported as such.
• attnres timing note: warm ~440 ms/step at batch 8 (vs moe’s ~657 ms — top-1 routing’s argsort per position per pass is NOT free), same ballpark as the 100k production control (463-515 ms).

Machine state at 01:21: GPU 3.9/16 GB (one train + desktop), RAM 39 GB available (20 GB in swap — watch, do not act), /home at 91 % (43 GB free; nightly ckpts ~40 MB each — monitored, the >95 % stop applies).

Merge for the night binary (done 01:2x, git-only, no build while the wave trains): main merged into wt/graph-trainer (merge c2e580b, clean, 49 commits — brings the A/B arms’ --set keys, the f16 gate, unguard). --graph-capture exists only in that branch; the arms only on main; the night binary needs both, so the gates run in the worktree and, if green, the branch merges back to main.

Phase 2 — verdicts (pending WAVE_DONE)

Control (yardstick, docs/guides/ab-testing.md): aux-ON 3-seed 6.387 / 6.314 / 6.329, mean 6.3433, range 0.0730 — §1.2 bar is the range. Wave runs carry the named ±0.05 systematic (retract-every 4 vs the control’s 1). Engagement check per arm: params= on the header line (mhc +6155 expected), fb=<ran>/<asked> for future-byte. Results → night-verdicts-2026-10-02.md

• benches/history.tsv rows.

Phase 3 — the night build (pending)

Gates, in order, all on the merged wt/graph-trainer, after WAVE_DONE:

1. Graph: the settling command from the gate doc §6.3 — CARGO_TARGET_DIR=/mnt/e43497ab-0ff2-45b4-b45f-28de3339a53e/dt-graph tools/build_lock.sh run graph -- cargo test -p dormouse-train --features cuda --test graph_seam_cuda -- --test-threads=1 --nocapture. The doc’s last fix (as_constant_float, 4849dcd) has never compiled. Three positive gates green → --graph-capture ships; red → no graph, blocker reported, NOT fixed tonight.
2. f16: 100-step probe fp32 vs --quant f16 (no graph — one variable): NaN episodes ⇒ f16 OUT (0 NaN required — the firewall skipping steps is not “clean” for a whole-forward numeric base change); ms/step and loss slope compared. Faster and clean ⇒ f16 ships.
3. Batch: 8/16/32 at ≥120 steps each (a 30-step run prints only the step-0 autotune timer — useless; --timers prints at step % 50), warm step-100 line read, best tok/s wins. Winning graph+f16/batch combination must not OOM (graph retains its working set — graph doc §6.6).
4. Auction: winning arms only (--set use_X=true), ties deleted.

Phase 4 — the overnight run (pending)

Real corpus + eval tail, --preset small, night flags per Phase 3, aux at preset defaults (jepa 0.05 + KoLeo, dspark 0 — the control recipe), --retract-every 4, 20 000+ steps, systemd-run --user --scope -p MemoryMax=40G, no --detach, log ~/logs/night_2026-10-02.log, ckpt night_20261002. Monitoring every 30 min: ms/step, NaN, loss, disk. Loud stops: NaN not extinguished by the firewall / ms/step ×2 / /home >95 %.

Baselines for the morning comparison: 100k production warm steps 463-515 ms (benches/history.tsv, first-100k-production row, pre-graph pre-f16 binary), batch ladder warm 244/440/826 ms at batch 8/16/32 (2026-09-29, attention backward absent then), L = 21 434 launches/step, graph promise 4-8× (arithmetic, unmeasured), f16 GEMM 40 TFLOP/s vs fp32 8.2-8.9.

Morning operator — Phase A (03:10-…)

fb needed nothing: all three wave-3 fb runs completed fresh (02:17-03:07, last-segment check: step-0 ce 6.141-6.150, params=9394318, no resume banner): 6.406 / 6.391 / 6.463 → mean 6.4200, +0.150 above the bar → TIE → deleted (table updated in night-verdicts-2026-10-02.md). The “4 runs without BEST” premise was stale by 20 minutes — only attnres was actually open.

• attnres_s2’s death is the 02:05 disk incident (above), not an arm defect.
• attnres_s3’s wave-3 “run” was a no-train resume of wave 2’s checkpoint (its 6.406 is prior evidence, the same number wave 2 printed) — so the arm’s verdict rests on s1 alone until the fresh re-runs land.
• Fresh re-runs launched (03:12, no --detach/--guard, the unguard rules): ab2000_attnres_s2r (fresh confirmed: two candidate refusals, one header, no resume banner, params=9200526, step-0 ce 5.575, warm 482 ms/step), then ab2000_attnres_s3r sequentially.
• Disk hygiene: the DELETED arms’ checkpoints (situ/mhc/moe — §1.2 deletions and the failed arm; logs + .config.toml snapshots kept) removed 26 files / 2.6 GB → /home 92-93 %, 40 GB free. Their .bin weights are re-generable in 20 min from the recorded flags and would never be used again; the evidence they carry (params, verdicts) lives in the logs and this document.

Phase C1 — the graph gate ran and is RED (03:55-04:05)

The §6.3 settling command ran on the free card: 1 of 4 gates green → --graph-capture ships in NOTHING, wt/graph-trainer stays unmerged (verdict + per-gate table written into the doc’s new continuation section, committed on the branch as 8dd8398). The two red positives are one finding: 6 replays leave the parameters bit-identical to the capture step — a window that excludes an out-of-place optimizer cannot train the model. The negative gate has also become unstable (illegal-address crash instead of the clean 1-10 relative signature it printed on 01.10), which goes to its owner with the rest. The 4-8× launch-overhead promise of §6.1 remains untested end-to-end; the night run ships fp32 launches, and the 21 434 launches/step figure is what the next launch-bound attack must beat.

Phase C2/C3 — f16 probe and batch ladder (03:55-04:12, main af9def1 binary)

All probes: 150 steps, --preset small, batch/quant as noted, --no-engram, --retract-every 4, aux at defaults, real corpus; warm step-100 --timers line is the reading. Logs ~/logs/probe_*.log.

mode	ms/step @100	tok/s	ce@100	NaN
b8 fp32-masters (auto factors = Fp8)	465	8 807	3.846	—
b8 --quant fp16	441	9 288	3.846 (bpb 5.549 vs 5.548)	0
b16 fp32-masters	894	9 166	4.090	—
b16 --quant fp16	930	8 808	4.097	—
b32	OOM at step 0 (failed to reserve 134217728 bytes)	—	—	—

• f16 ships for the night run — but only at batch 8. At b8 it is −24 ms (−5.2 %, the forward 185→162 ms — the f16 GEMM showing up where the 69433d4 gate said it would) with a loss slope identical to the fp32 arm and zero NaN/masked lines. At b16 the win inverts (930 vs 894 ms, the backward 504→534), so --quant fp16 is recorded as a batch-8 mode, not a global one. One 150-step probe per cell, single sample — the 4 % b16 inversion is within single-run spread; whoever campaigns f16 harder should re-measure with repeats, not extend the claim.
• Batch 32 does not fit with aux on — the JEPA teacher’s second forward is what the Sept 29 aux-off ladder did not pay. Loud OOM at step 0, no data, no retry.
• Night-run mode: batch 8 + --quant fp16 (9 288 tok/s, +5.4 % over the b8 fp32 arm). It also keeps the control family’s window (81 920 B) and batch, so its curve stays comparable to the 100k baseline and the wave. Steps set to 44 000 ≈ 5.6 h at the measured pace (start 04:18, finish ≈ 09:55).
• Graph: excluded (Phase C1). Arms: none (Phase B). The night build is therefore: control recipe + f16 — one execution-mode variable against the 100k baseline’s fp8-factors/batch-8; the morning report reads ms/step and loss against that row.

Phase D — the overnight run (launched 04:18)

systemd-run --user --scope -p MemoryMax=40G, no --detach, no --guard: --preset small --batch 8 --seq-len 512 --no-engram --retract-every 4 --steps 44000 --quant fp16 --eval-every 500 --timers, seed 1, log ~/logs/night_2026-10-02.log, ckpt night_20261002. Header verified: params=9197454 (control count, no arms), quant format: Fp16 (16 bits). Monitoring: ~/logs/night_mon.log, 30-min cadence, loud stops wired (5 consecutive nan ce lines / ms ≥ 2× warm base / /home ≥ 95 % → kill + report; monitor hands over at 10:00). Baselines for the morning read: 100k run warm 463-515 ms (fp8 factors), this run’s probe 441 ms, best held-out on record 5.545 @ 86k (100k curve), 4.997 (invalid-era caveat in §3.1).

Phase D — RESULT (finished 09:16, self-completed, no loud stops)

44000/44000 steps, 0 NaN in the whole log, 0 refusals, finished on its own 44 min before the 10:00 handover (04:18 → 09:16 ≈ 4.97 h; sustained ~8 700 steps/h ≈ 0.414 s/step including evals+checkpoints; warm 440 ms). Monitor readings every 30 min: ms 440-445 against base 440 (never ×2), /home ≤ 93 %, nan_lines=0 every reading (~/logs/night_mon.log). Artifacts: checkpoints/night_20261002{.bin, .best.bin,.config.toml} + sidecars; log ~/logs/night_2026-10-02.log.

Held-out: BEST bpb 5.908 @ step 36500, final-window 81 920 B; train CE best 2.512. Paired against the 100k production baseline (SAME seed 1, same batch/window/schedule-shape — the one differing variable is the factor quant, fp16 vs fp8), matched-step evals:

step	night (fp16)	100k (fp8)
500	6.469	6.480
5000	6.483	6.323
30000	6.119	6.111
36500	5.908	~5.95
42000	6.108	5.705
86000	—	5.545 (its best)

Reading, stated with its limits: at matched steps the two quants interleave inside the eval-to-eval noise (±0.1-0.2 on this window — visible in BOTH curves); fp16 is not a quality regression, and it is −5 % ms/step. The night run’s 5.908 is not comparable to the 100k run’s 5.545 (half the steps, cosine fully decayed at 44k vs 86k-to-best). What the morning inherits: a clean 44k-step f16 checkpoint whose curve says the f16 mode can carry production steps, and an open question — whether the fp16 advantage survives at 100k steps — which needs exactly this run again at --steps 100000 (12.6 h at the measured pace).

Morning report — one screen

• Verdicts (§1.2, bar 6.2703 = control mean 6.3433 − spread 0.0730): every arm ties, all deleted. moe 6.3377 · mhc 6.3653 [prior] · attnres 6.3300 fresh (6.191/6.346/6.453 — s2r/s3r re-runs landed 03:31/03:52) · fb 6.4200. situ: never ran (CUDA OOM, arm defect for its owner). The night build ships no arms.
• Graph: NOT in the build — 1/4 gates green; the red pair says a window that excludes an out-of-place optimizer cannot train (params bit-identical across 6 replays). Blocker in the graph doc’s continuation; branch unmerged, pushed as wt/graph-trainer.
• f16: in, at batch 8 only (441 vs 465 ms, slope identical, 0 NaN; at b16 the win inverts, b32 OOMs on the JEPA teacher). Ladder tok/s: b8+f16 9288 > b16+fp32 9163 > b8+fp32 8809 > b16+f16 8809.
• Night run: 44k steps, 440 ms warm (vs 463-515 in the 100k fp8 baseline), ~9 200 tok/s sustained, BEST 5.908 @ 36500, 0 NaN.
• Deferred: graph v2 (optimizer-in-window rewrite — owner decision), f16-vs-fp8 at 100k steps (12.6 h run), the attnres variance question (range 0.262 = 3.6× control), situ’s OOM, an ENOSPC pre-flight df before checkpoint saves, ~80 GB of landed worktree targets.