Skip to content

Loud failures: no silent data, no silent skips

Pretrain v21 trained on a constant b'x' stream: an unreadable corpus root produced an empty file list, and refill() silently fabricated filler bytes instead of failing. Train CE collapsed to 0.11 while everything looked like learning; only the held-out eval (a separate stream) exposed it. The same silent-failure shape existed elsewhere: a grad-shape mismatch skipped the host-table update instead of failing, a short read was space-padded, a v1 sidecar was reported and then discarded, max_ortho drift and the drift check were the only loud mechanisms.

Decision: NASA P10 Rule 5 (docs/research/2026-09-25-nasa-burn-rust-practices.md) — assertion density >= 2 per non-trivial function on average, and every assertion failure routes to a named recovery action (a startup config error is a hard error with the offending path named; the named recovery for a mid-run failure was the --guard checkpoint-and-resume, removed 2026-10-02 after its detached parent exited 0 and orchestrators read failure as success — docs/reviews/unguard-2026-10-02.md; the recovery today is a non-zero exit and an operator resume with the same --ckpt-name, with the NaN firewall unconditional in the loop). Data that does not exist stops the run; it is never synthesized. Shape mismatches stop the run; they are never skipped. Backwards-compat fallbacks that discard trained state (sidecar relayout) are hard errors naming the escape (delete the file or pick another --ckpt-name).

Landmarks: collect_files/from_files asserts + corpus size floor (commit 9605896), the refill dry-assert and short-read assert (data/lib.rs), host-grad shape assert and sidecar hard error (train/lib.rs), dead-field validation removed rather than kept as false confidence (docs/archive/audit-2026-09-25.md §3.10).