unguard: the --guard wrapper is deleted, the NaN firewall stands alone

Date: 2026-10-02 · Lane: unguard (owner brief after the evening incident of 2026-10-01) · Commits: 1949c39 (code), d0df2b2 (live docs), this one (findings).

The incident, in one paragraph

--guard re-exec’d train on NaN/panic: wait 30 s, re-exec (through the pinned image under target/release/guard-image/), resume from the last checkpoint, capped at 3 restarts. Under --detach the launcher parent exits 0 the moment it forks, so every orchestrator that keyed on the exit code read a crashed-and-restarting run as a finished one and launched the next arm. Restarted runs and new arms accumulated on one GPU — fifteen concurrent trainings, OOM. The wrapper did not just fail to recover; it manufactured the exact “two GPU processes at once” condition that §1.5 exists to prevent, while reporting success.

What was removed

All of it lived in crates/dormouse-cli/src/bin/train.rs (the CLI binary, not the trainer library):

removed	where it was
the --guard clap flag	train.rs:196-199 (old numbering)
the guard-image pinning block: copy self → target/release/guard-image/train, DORMOUSE_GUARD_EXE, unconditional re-exec through the image	train.rs:278-315
DORMOUSE_GUARD_RESTARTS restart chain: catch_unwind, resumable probe, 30 s sleep, re-exec with original argv, the cap at 3	train.rs:361-421
the 196 MB pinned-image artifact	target/release/guard-image/ (untracked; deleted; no process had its inode open — checked /proc/*/exe first)

Net: −109 / +11 lines. main is now linear: parse → resolve → detach → log redirect → autotune env → execute → on Err print and exit(1). A panic takes the default unwind path: message on stderr, exit 101. Both exits are loud and nothing lingers. Passing --guard to the new binary is a clap error (error: unexpected argument '--guard' found) — an unknown flag refuses loudly, per the no-silent-surface rule; no orchestration can half-work.

What became unconditional: nothing had to change

This is the load-bearing finding of the lane: the NaN firewall was never behind the flag. --guard was consumed entirely in the CLI main; it never reached TrainCfg (grep: crates/dormouse-train/src has no guard field, only prose comments). The firewall — mask_fill of a non-finite loss to 0 on device, then sanitize_grads zeroing every non-finite gradient on device, then the step skip — runs on every step of every run, flag or no flag (crates/dormouse-train/src/lib.rs:1265-1287, gate test nan_firewall_makes_a_nonfinite_loss_a_noop, lib.rs:~2898). Deleting the wrapper changed nothing about in-loop protection; it only removed the process-level restart that made failures lie about themselves.

The deferred lib.rs diff: none needed

The brief’s hard constraint anticipated that flag parsing / re-exec might live in crates/dormouse-train/src/lib.rs and ordered a ready diff if so. They do not: the flag parse, the image pinning and the restart chain were all in crates/dormouse-cli/src/bin/train.rs, which no other lane holds. The lib.rs guard hits are unrelated prose (the firewall’s own comments, the stress monitor’s “nothing to guard”, test names). lib.rs was not touched — zero conflict with the live graph lane.

Gates

gate	result
cargo build --release -p dormouse-cli --features dormouse-train/cuda (build_lock unguard; separate target dir, see deviation)	GREEN — Finished release profile in 13m 15s, 2026-10-02 00:52
cargo test -p dormouse-train --lib	GREEN — 52 passed; 0 failed (includes nan_firewall_makes_a_nonfinite_loss_a_noop), 2026-10-02 00:52
1-step smoke, new binary, no --guard (preset nano, batch 1, s128, real corpus)	GREEN — rc=0; step 0 ce=5.535, checkpoint saved, log format unchanged (the firewall line prints only when it fires — it did not need to)
negative: --guard refused loudly by clap	GREEN — error: unexpected argument '--guard' found, exit 2, before any device init
grep -rn "guard-image|--guard" README.md AGENTS.md tools/ docs/guides/ → 0 in live docs	GREEN — 0 matches (ram-guard, the systemd service, is unrelated and stays)
python3 tools/check_doc_refs.py green	GREEN — 0 named paths missing

Two self-inflicted detours the gates caught, recorded because both are the kind of confidence that costs an hour: (1) with CARGO_TARGET_DIR=<dir>/dt-unguard the binary lands at dt-unguard/release/train, not dt-unguard/target/release/train — the first smoke attempt failed with 127 on the wrong path twice, including after a no-op 1.79 s “rebuild”; nothing was ever deleted, the file was one directory level off from where I looked. (2) The first smoke script died to my own pkill -f matching the shell’s own command line (§2.4 item 6, again).

The grep gate: zero in README.md, AGENTS.md, CONTEXT.md, CONTRIBUTING.md, tools/, docs/guides/, docs/adr/0011-loud-failures.md, docs/glossary.md. Deliberately left untouched — dated documents are history, not knobs (the same rule that keeps goal.md’s tails): docs/reviews/ab-wave-2026-10-01.md:71, docs/reviews/2026-09-29-engram-jepa-dspark-review.md:1256, docs/research/2026-09-25-nasa-burn-rust-practices.md:198, docs/research/2026-09-27-adaptive-depth-safe.md:521. The glossary guard entry is now a tombstone pointing here, so a reader who hits --guard in a dated doc learns in one line that it is gone and why.

Deviations

• Build target dir moved off the brief’s literal path. The brief said CARGO_TARGET_DIR=/mnt/dt-unguard; /mnt is on the root filesystem (464 G, 50 G free, 90 % full) and a cold CUDA release build is a multi-GB artifact (the main target/ alone is 41 G). §2.5 names a full disk mid-build as a real failure mode. The dir is /mnt/e43497ab-0ff2-45b4-b45f-28de3339a53e/dt-unguard (the data drive, 596 G free) — same isolation property, which is the point: the live ./target/release/train inode is never touched, the running wave keeps its binary. The dir (2.8 G) is deleted after the gates.
• doc-refs ordering: d0df2b2 (live docs) landed a few minutes before this findings file existed, so check_doc_refs was red in that window — five dangling refs to this path. Green from this commit on.
• The smoke waited out the whole A/B wave. The wave’s inter-arm gaps are shorter than any safe double-check (the next arm launches as the previous process dies), so the smoke could not be inserted between arms and ran at the first clean window, after attnres_s2’s slot. §1.5 was never violated: the smoke started only on two consecutive train-free reads 20 s apart.

What the wave looked like from this lane (context, not findings)

The seed wave ran through this lane’s whole window: situ s3 died at step 200 (device error (loss is not a scalar), pre-unguard binary — not this diff), attnres s2 died at step 1000 (checkpoint save failed: No space left, root fs hit 0 free around 01:50 and recovered to ~46 G by 03:00 — cause not this lane’s; a 2.8 G build dir on the data drive cannot fill the root fs). Both are the wave owner’s follow-ups, listed at the bottom.

Follow-ups (not this lane’s)

• The wave’s situ seed-3 arm died at step 200 with train failed: step 200: device error (loss is not a scalar) on the pre-unguard binary (00:20:34, ~/logs/ab2000_situ_s3.log) — pre-existing, unrelated to this diff (this lane touches no loss path), but the owner should know the situ arm has 2 of 3 seeds.
• The wave’s attnres seed-2 arm died at step 1000 with checkpoint save failed: No space left (~01:50, ~/logs/ab2000_attnres_s2.log): the root filesystem hit 0 free mid-save and recovered to ~46 G by 03:00. Not this lane’s bytes (the build dir lived on the data drive), but the second wave death of the night — both are the wave owner’s follow-ups.
• docs/reviews/ab-wave-2026-10-01.md’s control recipe still names --guard in its launch line; future wave scripts copied from it will fail at clap, loudly, which is the correct failure — but the next wave script should be written without it.