BPB honesty audit — are all bytes counted, the divisor, the target shift, the train/held-out leak

Lane: bpbaudit, worktree wt/bpbaudit off 599cf0f. Date: 2026-10-04. Suspicion №1 from .bulba/goal.md:502: «все ли байты учтены, делитель, сдвиг target, утечка train/held-out», and the overfitting question (в): «dormouse ПЕРЕОБУЧАЕТСЯ: train-bpb 2.569 vs held-out 5.969».

CPU-only. No GPU was touched (a bar-test holds the card). Nothing was rewritten — this is an audit. The two conventions it found are pinned by a new gate, not fixed: crates/dormouse-train/tests/bpbaudit_seam.rs (4 tests, green).

0. Verdict

#	point	verdict	gate / citation
1	BPB = ce / ln2?	HONEST	`train/src/lib.rs:36` — `ce / std::f32::consts::LN_2`; pinned by `bpb_is_ce_over_ln2`
2	target shift = x[q+1]?	HONEST (one carried convention)	shift is exactly one byte at every call site (`lib.rs:1383`, `lib.rs:1873`, `byteflow.rs:355`); the wrap is pinned by `target_shift_is_one_byte_with_wrap`
3	all bytes counted? mask / padding / BOS?	HONEST	no mask in pretraining; divisor is `t` then `b` (`loop_block.rs:1209`, `:1214`); eval mean over all `b*t` (`lib.rs:1918`); short read panics (`data/lib.rs:826`) — no padding exists
4	eval window = eval-batches × batch × seq_len? rewind?	HONEST	`ev.rewind()` before every eval (`lib.rs:1832`); `bytes = n * batch * seq_len` (`lib.rs:1939`); pinned by `eval_window_rewind_and_mean_divisor`
5	train/held-out leak? dedup? domains?	HONEST (one caveat)	path guard (`data/lib.rs:549-558`) + positional carve with straddlers dropped (`filter.rs:1243-1262`) + global exact dedup (`filter.rs:1164-1175`). Caveat: domain balance is not enforced — see §5
6	byteflow path: mask / shift / divisor?	HONEST (same wrap)	`byteflow.rs:355-360` (shift), `:209-214` (mean), `:243` (rewind), `:252-257` (mean), `:267-268` (bpb + scored)
7	is the held-out number honest? (train 2.569 vs held-out 5.969)	HELD-OUT HONEST; train-line `bpb=` is NOT pure CE	the logged train `ce=`/`bpb=` includes the aux objective (`lib.rs:1529-1535`, `:1601`, `:1799`); the held-out `bpb=` is pure next-byte CE. Pinned by `train_logged_loss_is_ce_plus_aux`

The instrument is honest where it matters most: the held-out BPB is a pure next-byte CE over a fixed, rewound, leak-free window, with every position in the denominator. Two carried conventions distort the train-side number and the absolute BPB, both in the safe direction for the overfinding claim — and both are documented below with their magnitude, because a convention nobody can find in the diff is a defect wearing a documentation costume.

1. The divisor: `bpb(ce) == ce / ln 2` — HONEST

crates/dormouse-train/src/lib.rs:36:

pub fn bpb(ce: f32) -> f32 { ce / std::f32::consts::LN_2 }

One function, two call sites (lib.rs:1799 train, lib.rs:1938 eval), the same divisor. A CE in nats per byte becomes bits per byte. The uniform anchor: a 256-class model with CE = ln 256 scores exactly 8.000 BPB. Pinned by bpb_is_ce_over_ln2, including the uniform case.

2. The target shift: exactly one byte, with a carried wrap — HONEST with a convention

The construction is one iterator chain, four call sites:

bytes.iter().skip(1).chain(std::iter::once(&bytes[0])).map(|&b| b as i64)

train step: lib.rs:1383 (built from the prefetched batch — the DSpark lesson is applied; the comment at lib.rs:1375-1378 names the failure it prevents),
eval: lib.rs:1873-1878,
byteflow: byteflow.rs:355-360,
warmup / quant-check: lib.rs:1163, :1179, :1222.

The shift itself is correct: targets[q] == bytes[q+1] for every q < N-1. There is no DSpark-style off-by-one in the CE path — the head is not fed the byte it was predicting. The DSpark aux window is separately teacher-forced from the consumed ids, not the labels (aux.rs:355-363), which is the fix AGENTS §3.3 describes.

The convention: the last position of each batch targets the batch’s own first byte (bytes[0]), not the next byte in the stream (which is the first byte of the next batch). So 1 of N = batch * seq_len positions per batch has a circular target.

Magnitude, honestly bounded:

The target bytes[0] is in the context of the last position (causal attention sees the whole prefix), so the model can learn the copy convention — the position is artificially easy, deflating CE by up to ce/N.
If it cannot, the position is unpredictable, inflating CE by up to (ln 256 - ce)/N.
At batch 2 / seq 512 (N = 1024): |Δ| ≤ 8.0/1024 ≈ 0.008 BPB. At batch 10 (N = 5120): ≤ 0.0015 BPB.

Why this is not a verdict-killer: the convention is identical in train and eval, so the train/eval comparison is fair; and it is batch-size dependent, which is a comparability wrinkle across batch sizes (the same class as the §2.6 window issue), not a dishonesty between the two numbers on one line. It is already pinned in sft_smoke.rs:177-189 (“The trainer’s own label construction, copied … with the wrap”) and now in target_shift_is_one_byte_with_wrap.

Fix, if ever worth it: next_bytes returns N+1 bytes and the last target is bytes[N]; or, since the next batch is already prefetched, take the last target from nb[0]. One line. Not done here — audit, not rewrite.

3. All bytes counted — HONEST

Train CE (loop_block.rs:1142-1214): per-iteration log_softmax.gather(target).neg() → [b, t]; with loss_mask = None (the pretraining path) the row divisor is t (:1209) and the batch divisor is b (:1214). Every position counted, none dropped, no BOS, no padding — the stream is raw contiguous bytes.
Eval CE (lib.rs:1918-1923): log_softmax(eflat).gather(etgt).neg().mean() — mean over all b*t positions, no mask.
No padding exists to hide in: next_bytes asserts bytes.len() == need (data/lib.rs:826-831); a short read is a LOUD panic, not a silent pad. The “falls back to padding” doc comment at data/lib.rs:761 describes a path the assert makes unreachable.
The SFT arm (loop_block.rs:1204-1210) has a loss_mask that divides by the mask’s own count — honest by design (keeps the scale of a fully-scored batch), and it is not the pretraining path.

4. The eval window — HONEST

ev.rewind() before every eval (lib.rs:1832), with the comment naming the 2026-09-27 measurement (same checkpoint scoring 6.443 vs 6.551 from stream position alone). rewind itself asserts the ring has not drained (data/lib.rs:780-789), so the “fixed window” guarantee cannot silently void.
The window is eval_batches batches of batch * seq_len bytes (data/lib.rs:800-801), and the byte count printed on the eval line is n * batch * seq_len (lib.rs:1939) — the same number the ece mean divides by. Pinned end-to-end by eval_window_rewind_and_mean_divisor (real stream, real model, rewind restores the same first batch).
The eval forward is model.valid() (no autodiff — lib.rs:1838), full depth (set_loop_depth(None), :1844), and passes the same memory arm the training step runs (:1906-1912) — the 7adda92 fix, with the engram=/ bf= counters on the line as the check.

One wrinkle (diagnostic, not a held-out claim): --eval-depths (lib.rs:2045-2099) does not rewind between depths, so each depth scores a different window (the stream advances). The depth curve is a deliverable for depth claims, not a held-out number, so this is noted, not flagged.

5. Train/held-out leak — HONEST, with a domain caveat

Three independent mechanisms, all verified in code:

Path guard (data/lib.rs:549-558): train_and_eval canonicalises both roots and refuses when either contains the other — the ADR-0010 rule, with the failure named (training on the held-out bytes).
Positional carve (filter.rs:1231-1262): docs are routed by start offset; the eval tail is the last --eval-tail-mb MiB (filter.rs:1348-1351); docs straddling the boundary are dropped entirely (:1245-1248). No byte overlap between head and tail by construction.
Global exact dedup (filter.rs:1164-1175): one shared Pipeline for both regions; a 128-bit doc hash seen in the head drops the tail copy (“no leak in either direction”, :1228-1230). The tail goes through the same quality threshold as the head (on_doc is the only gate).

The caveat — domain balance is not enforced. The eval is a positional tail carve: whatever domains sit at the end of the corpus file are what gets scored. If the corpus is domain-ordered (by source, date, language), the held-out window is domain-skewed relative to the training head, and the held-out BPB is not a number for “the corpus” but for “the tail”. Nothing in code checks this; it is a data-construction property. The anchors.rs:22-35 comment already records that the in-corpus --holdout split and the trainer’s eval window were never comparable. Measure the domain mix of eval_tail.bin before quoting the held-out BPB as a corpus-level number. This is a caveat on interpretation, not a defect in the instrument.

6. The byteflow path — HONEST (same wrap)

byteflow.rs:

shift: batch_tensors at :355-360 — the identical skip(1).chain(once) chain, so the same §2 wrap convention (1 of N positions).
train CE: :209-214 — sum_dim(1).div(t).sum_dim(0).div(b), mean over all positions, no mask.
eval: ev.rewind() at :243; ece = log_softmax.gather.neg().mean() at :252-257; scored = n * batch * seq_len at :268; crate::bpb(ece) at :267 — the same divisor, the same window formula.
The byteflow train CE is pure (aux=none, :156): no JEPA/DSpark term is added. So byteflow’s train-line bpb= is comparable to its held-out bpb= (modulo the wrap) — which makes the dormouse path’s aux inclusion (§7) a divergence between the two trainers, not a universal property.

7. The overfitting question — the held-out number is honest; the train-line `bpb=` is not pure CE

The held-out 5.969 is a pure next-byte BPB: fixed rewound window (§4), every position in the denominator (§3), no aux (the eval forward passes targets = None, so aux = None and the in-loop rec is zero — pinned by train_logged_loss_is_ce_plus_aux), leak-free by construction (§5). It is a number for the network that produced it.

The train-line ce=/bpb= is the FULL objective, not the pure CE. lib.rs:1529-1535:

let mut loss = model.loss::<Backend>(rec_ce);   // pure CE
if let Some(a) = aux { loss = loss + a; }        // + JEPA + KoLeo + DSpark + fb + moe_lb

and loss_log (that sum) is what gets read back (:1601) and printed as ce= / converted to bpb= (:1799). The aux terms are all non-negative (model.rs:636-698: JEPA squared-error, DSpark BCE, KoLeo, future-byte, MoE load-balancing), so:

pure train CE = logged ce − aux        (aux is printed on the same line)
pure train bpb ≤ logged bpb = 2.569

Consequence for the overfitting claim: the true gap held-out − pure-train is larger than 5.969 − 2.569 = 3.4. The overfitting verdict is safe — the logged numbers understate it — but the two fields on the train line are not the same quantity as the held-out field, and a reader who subtracts nothing is comparing (CE + aux)/ln2 to CE/ln2. The aux= value is printed on the same line, so the information is present; the bpb= label is what misleads.

Fix, if ever worth it: log rec_ce (pure) as ce= and the objective as obj=, or print bpb= from the pure CE and bpb_obj= from the objective. Two lines. Not done here — audit, not rewrite. The A/B program (§1.2) is unaffected: arms are judged on the held-out BPB, which is pure.

8. Gates

crates/dormouse-train/tests/bpbaudit_seam.rs — 4 tests, CPU-only, green:

test	pins
`bpb_is_ce_over_ln2`	the divisor; uniform = exactly 8 BPB
`target_shift_is_one_byte_with_wrap`	the one-byte shift at every call site; the wrap convention
`eval_window_rewind_and_mean_divisor`	window = `eval_batches × batch × seq_len`; rewind restores the same window; the mean’s divisor = the scored bytes
`train_logged_loss_is_ce_plus_aux`	the logged train loss is `rec_ce + aux`; the eval forward carries no aux

cargo test -p dormouse-train --test bpbaudit_seam — 4 passed, 0 failed.

9. What to fix, ranked

Train-line bpb= includes aux (§7) — the only finding that changes what a reader concludes from a log. Direction is safe (overfitting understated), but the label is wrong. Fix: print pure CE as ce=, objective as obj=.
Domain mix of eval_tail.bin unmeasured (§5) — the held-out BPB may be a tail-domain number, not a corpus number. Fix: measure, then quote the domain with the number.
The wrap target (§2) — ≤ 0.008 BPB at batch 2, batch-size dependent. Fix: one line (nb[0] as the last target). Lowest priority: it is deliberate, pinned, and identical in train and eval.
--eval-depths does not rewind between depths (§4) — diagnostic only. Fix: rewind before each depth if the curve is ever quoted as a held-out claim.

None of these touch the held-out number, which is the one the A/B program rests on.