Skip to content
- Every held-out BPB from a run that had the in-VRAM Engram ON. Until
7adda92, the eval forward passed hashed_ids = None unconditionally
(train/src/lib.rs, the - line of that commit’s diff) while the training
step passed real keys. eval_rows is Some only when the host offload is
live (lib.rs:1361-1374), so on the in-VRAM path loop_block took its inert
branch and returned zeros: the held-out number was a measurement of a
different network than the one being trained. --eval-depths had the same
defect and is fixed in the same hunk.
- It bites exactly when
use_engram = true AND engram_ram = false — the
in-VRAM Engram arm. It cannot bite when use_engram = false, because a
forward with no memory in it is the correct forward for a model with no
memory in it; and it cannot bite under --engram-ram, where eval_rows was
already Some and the host rows were passed. That scoping is the whole
finding: this is a retracted arm, not a retracted history.
- Touched: 2 of the 12 config snapshots in
checkpoints/ —
engram_ce (batch 2, depth 2, 25 000 rows) and small12 (batch 10, depth
4). engram_ce’s 6.453 at step 2000 is invalid, and so is every other
number on that curve. For small12 the config is in the affected class but
no eval line for it is on record in ~/logs/, so nothing is retracted
that was not printed; it is listed so nobody reuses its snapshot believing
the memory arm was scored. The other 10 snapshots are all
use_engram = false.
- “The Engram arm lost by 1.46 BPB” is not a verdict, and never was. The
comparison was
engram_ce’s 6.453 against nokda_ce’s 4.997, and the
first of those is a memory-enabled training run scored by a
memory-disabled evaluation of itself. Note what was not wrong with it:
both runs are batch 2, so both scored the same 20 480 B window at the same
depth 2. The window matched. The eval defect is the whole of the
failure — there is no second reason to discount it, and no second reason
to rescue it either.
- Not touched: the 4.997 at step 6500.
nokda_ce.config.toml has
use_engram = false, so nothing was missing from that eval — the None was
the right answer. It also has use_kda = false, so it is clean of the
second defect below. It is the one held-out number in the archive that is a
number for the network that produced it, and it stands, with the caveats in
§3.1 (depth 2, a --no-kda model, a 20 480 B window, and best-of-an-
overfitting-curve).
- The evidence is a counter, not a reading, and the counter is newer than
the defect:
engram=<rows>/<arms> was added to the eval line in the same
commit as the fix (7adda92), so no pre-fix log in ~/logs/ carries the
field at all — train_nokda.log, train_kda_full.log and
train_engram25k.log all end in muon_skipped=0/0 with nothing after it.
That the pre-fix eval read 0/0 is c4214ad’s assertion, not something
the logs show. What the logs do show is the post-fix reading on hardware:
a 20-step run with the Engram on printing engram=4/4 — 4 rows over 4 arms
for --eval-batches 2 at depth 2 (c4214ad). Going forward the field is the
check: engram=0/<n> means the eval just ran a memory-disabled forward.
- The commit that carries the fix is
7adda92, whose message is about
the best-checkpoint and does not mention it. b3d6914’s message does
claim the eval fix and its diff does not contain it; c4214ad is the
no-leak enforcement (ByteStream::train_and_eval). Cite 7adda92. A
commit message that describes a fix it does not carry is the ADR-0020
failure in its purest form, and it is why the fix was hard to find.
- Every held-out BPB from a run with
use_kda = true. 8fa5d4c:
chunk_wy_forward_autodiff_s ran the forward on the bare backend and
wrapped the result in one hand-rolled autodiff node. OpsPrep::prepare
decides tracked-ness from the parents’ node refs, and under
BalancedCheckpointing — the trainer’s strategy — the op’s inputs are
checkpoint leaves, so the node came back UnTracked and its output was a
leaf. The tensor-ops fallback lives inside that node’s backward, so a leaf
means no gradient either way. The attention arm had no gradient for the
entire history of this project: it ran thousands of forwards and trained
nothing, and every other op built its own graph, which is why the loss curve
looked healthy and only the counter was wrong.
- The counter is in this box’s logs:
~/logs/train_kda_full.log prints
fused kda=1046/0 at step 500 and 3126/0 at 1500 — 3126 forwards,
zero backwards.
- So
kda_ce, kda_full, kda_smoke{,2..5}, noengram_ce, probe2 and
small12 all reported a network whose attention arm sat at initialisation.
The 6.351 in §3.1 is one of them (use_kda = true), which is the second
reason that row is not a depth-4 quality result.
- This also explains a speedup that was never a speedup: the fused
forward made attention look nearly free (+22 ms) precisely because it was
doing no backward work at all. A fused path that skips the backward is not
faster than a tensor path that runs it; it is a different program.
- The fix is verified to compile and NOT verified to train.
8fa5d4c’s
own message says the gradient-flowing check is the next step and is not
done; DM_GDN2_BWD_TRACE=1 should print ENTERED on ChunkWy::backward,
and that line appears in no run before it. Treat “the attention arm trains
again” as unverified until a run shows it.
- Cost of the working backward, as reported by the 2026-09-28 audit and
not reproduced here, with no committed log and no
benches/history.tsv
row: the tensor-op path at batch 8 measured 25.8 s/step against
3076 ms without the arm. If that holds, the A/B budget in
docs/protocols/AB-PROTOCOL.md (53 min per 2k-step run) is wrong by more than an
order of magnitude and the queue has to be re-costed before it is run. The
4628-era note that the same op costs 3628 ms fwd+bwd (README.md:77) is
a different shape and is not a substitute.
- Every parquet corpus result before
f6ab353. Source::read
downcast the top-level columns to StringArray and dropped everything else
with no counter. mix/qa nests its text (document: struct { html, title, url, tokens }), so the only top-level string was id. Measured on one 200 MB
shard asking for 8 MB: 21 811 B before the fix, 8 000 000 B after — and
the bar tool printed anchors either way (“unigram 3.544 / 5-gram 3.278” on
21 KB of UUIDs, 367× short). Every parquet run trained on a fraction of its
corpus and reported the loss curve as if it had not. A shard that yields no
text is now counted by name rather than vanishing.
- “fp4 + group 128, 100 steps, 0 NaN, convergence == fp32” (act quant) —
retracted twice over, for two independent reasons, and the second is the one
that makes the number meaningless rather than merely partial. (1) The
attention upgrade is unconditional —
ActFormat::attn() maps Fp4 -> Int(8)
and Int(b) -> Int(b.max(8)) — so --act-quant fp4 has never run 4-bit
attention; that verified the FFN path only. (2) 9b343d3: fp4 was not
e2m1 at all. Real e2m1 has no 0.75 (the grid is 0, .5, 1, 1.5, 2, 3, 4, 6),
the mantissa rule emitted 0.75 for a ∈ [0.625, 1), and the caller scaled
each block’s max onto 1 rather than onto the format’s max (6) — so only
{0, 0.5, 0.75, 1} of the eight magnitudes were reachable. That is a
~3-level quantizer wearing a 4-bit label, and 1.5…6 was dead code. Every
--act-quant fp4 number before that commit measured the 3-level thing. The
4 and 8 paths are untouched and pinned bit-identical
(int4_levels_are_the_whole_symmetric_range).
- All Gated Residual numbers — of which there are none. Worth stating
precisely, because “retracted” would over-claim:
use_gr = false in all
eight configs/*.toml, so no preset, log, checkpoint or A/B number is
invalidated. What was wrong is in 9b343d3: the write omitted the sigmoid
and the factor 2 of Eq. 33 while the comment above it wrote out the
equation the code did not implement; the read omitted the 1/nr that sits
inside the SiLU (Eq. 31), giving a gate saturated at init; and the readout was
taken from the state before the write, so GR was a depth-(iters−1) model
and at max_iter = 1 the entire block body was computed and discarded. The
report’s −0.026 and its zero-spike result cannot transfer in any case, because
the report puts one GR per attention and MLP sublayer of a 56-sublayer stack
and we run one weight-shared block recursively (§3.5). The arm has still
never been A/B’d, which is the debt this does not discharge.
- “fused is 1.7-2.0× faster than burn” (ADR-0003). Retracted in ADR-0009. At
flagship the whole-loop fused op measured 9.21 s/step against burn’s 7.05
— 1.3× slower (
docs/archive/research/2026-09-23-fused-flagship50.md:52-53). The
fused/ module and the DM_FUSED switch are now deleted; its kill
switch in ADR-0009 is moot. What survives under the name “fused” is the
library’s own fused kernels, which are live and counted.
- “the bool→float cast returns 0.0 for true on cuda” (memory.md:15). Wrong;
the miscount was
clone() aliasing plus a firewall that did not skip the
step (ADR-0016). One code comment still says it — see the glossary.
- Every train-CE reading from the PonderNet era. It measured λ-collapse, not
learning (ADR-0013).
- Every aux-vs-pure-CE A/B conclusion before 2026-09-27. The EMA teacher was
being fed the label sequence, so the JEPA target was wrong in every run since
the aux heads shipped. DSpark’s window shift is fixed (§3.3, 2026-09-29);
verify-tails then found two more defects in the same expression, both fixed
with gates (
docs/reviews/verify-tails-2026-09-30.md): dspark_loss did
not detach the confidence target (the acceptance head back-propagated
into its own target — DeepSeek’s rule is a detached c*), and the BCE was
spelled as the usual log(σ)+log(1−σ) pair while the correct stable form is
c·(−softplus(−z)) + (1−c)·(−softplus(z)); the original gradient fixture had
no gradients at all (asserted on zero tensors). Post-fix verdict that
stands: JEPA+KoLeo beats pure CE 3/3 seeds (d8062d1).
- The
max_ortho fallback story before 2026-09-04: the fp32 fallback fired
on every fresh run, so no pre-fix run actually used the factor-quant forward.
- The 2026-09-27
--no-kda 956 → 188 ms ablation: measured alongside a live
run, so it is a ratio, not an absolute. Step-time claims need same-shape
measurement on a quiet GPU. It is also the only step-time measurement of
removing the attention arm, and the arm was not training at the time, so it
does not price the arm the fixed one costs.
- Every A/B verdict in
docs/protocols/AB-PROTOCOL.md. Not one arm has been judged.
The instrument was wrong for the Engram arm (above), the attention arm in
every control was frozen at initialisation (above), and no two runs in the
archive share a window size. The queue is a list of experiments to run, not
results, and its cost line is void.
small = 7.5M / base = 12.2M params. Pre-date the 2026-09-27 memory
re-pricing; the measured numbers are in tests/preset_exec.rs and the README
table, and the 7.5M figure is still hardcoded as a test constant
(loop_block.rs:706) and asserted in a doc comment (schema.rs:130).- “the fused KDA op is 11.2x faster than PyTorch on fwd+bwd” (2026-09-28).
Only the FORWARD claim survives.
alloc_probe and both benches build their
device as Device::autodiff(...) = NoCheckpointing and enter the node with
B = Autodiff<..>, where the adjoint’s TypeId gate at
chunk_adjoint_cube.rs:399 returned None — so the tensor adjoint ran
and the measured fwd+bwd time contained it. Worse,
fused_chunk_verify.rs:132 (fused_op_grads_match_tensor_path_cuda)
compared that tensor adjoint against the tensor path, i.e. verified the
tensor adjoint twice, and the only caller of fused_chunk_backward in the
whole tree is bench_fused_bwd.rs:146 — bare tensors, inside a timer, result
discarded. The fused adjoint kernels have therefore never been numerically
compared to anything. A counter placed BEFORE that gate is what let the CUDA
gate test assert bwd > 0 and pass; it now sits after the gate (f737710),
so the test is deliberately red until the backward gate lands. Do not
quote a fused fwd+bwd number until the new gradient comparison is green.