moe-dev — instrument, protocol, capacity ladder, TSCT at scale

Date: 2026-10-03. Lane: moe-dev (branch wt/moedev). Commit of the work described here: the four commits below 98f98d7. Baseline: main @ 146df4c.

Nothing here is an A/B result. The GPU was held by night2e for the whole lane (protocol §1.5: one GPU process at a time), so no run happened. What exists is: an instrument with gates, a protocol that runs unattended, a measured capacity ladder, and a scale analysis. §1.2 is untouched by this lane — no arm has been judged, and the word “verdict” below appears only inside the script that will produce one.

---

1. The utilization instrument (deliverable 1)

Why it was missing

A top-k router that collapses produces a perfectly healthy loss curve. tsct-spec-2026-10-03.md §12 establishes that top-k does not dispatch — every expert is computed every iteration — so a collapsed router is not slower, not noisier and not visibly different in any loss number. The blend is still a valid convex combination either way.

The existing counter-measure is not a measurement. lb_aux is Switch’s load-balancing term; moe_lb_coef ships at 0.0, so the term is computed, weighted at nothing, and reports nothing about the distribution it exists to keep balanced. tsct-spec §10 item 5 (“does the router learn to specialize? — utilization-entropy”) therefore had no instrument behind it at all. That item is what this section closes.

What it is

moe::util_stats(mask, k) -> Tensor<1> of length n_experts + 2, entirely on device:

slot	meaning
[0..e]	share_e — fraction of tokens whose top-k included expert e
[e]	H — Shannon entropy of that share, normalized by ln e, in [0, 1]
[e + 1]	dead — experts that received zero tokens

Two design decisions, both argued in the code:

Entropy of the HARD share, not of P. The share is what a real dispatch would bill (the thing §12 says must be built) and what “dead expert” is defined against (f_e == 0). P — the mean router probability — is what lb_aux already reports and is a different number.

Normalized by ln e, because that is the whole point. The capacity ladder moves n_experts from 2 to 16. Raw entropy is ln(e) at uniform, so a raw H reads 0.69 on a healthy 2-expert router and 2.77 on a healthy 16-expert one, and comparing across the ladder would say nothing about either. On the normalized scale 1.0 is uniform and 0.0 is one-hot, at every e.

Wiring and cost

forward_full_state records one util_stats per routed iteration, averages them the same way lb_terms is averaged, and hands the result to a thread-local seam — the shape mixture_probe already uses for the per-iteration blend. The trainer takes it at log cadence and prints:

moe=[0.562,0.438,0.000,0.000] H=0.494 dead=2

One host read per log step and nothing else (§1.3 — no into_data in the hot path, no per-step scalar read). The recorded tensor is detached, and that is load-bearing rather than tidy: an un-detached statistic would sit in the thread-local holding a reference into the previous step’s autodiff tape until the next forward overwrote it.

ADR-0019 COUNTED in the strongest form available. The field is absent unless the routed branch ran, so there is no state in which the field exists and means nothing. A collapsed router prints H=0.000 dead=4, which is a number; an arm that never ran prints nothing. Those are distinguishable, which is the requirement.

Gates — 4 new, in 2 files

test	what it would fail on
util_entropy_is_one_at_uniform_and_zero_at_collapse	either endpoint, or the shares, drifting
util_counts_a_partially_dead_router	H checked against the host entropy of (4/6, 2/6, 0, 0), not against a range
util_entropy_falls_when_the_router_learns_to_specialize	the sign of the metric
the_routed_arm_records_a_utilization_and_the_dense_control_does_not	the SEAM — routed records, dense records nothing, take clears

The third is the one the instrument rests on. It trains a router: 32 rows × 4 experts, the objective is the mean pre-top-k probability of two designated poison experts, gradient ascent on the logits. Measured:

H 1.0000 -> 0.4943,  dead 0 -> 2,  shares [0.5625, 0.4375, 0.0, 0.0]

A metric that rose with specialization would pass every other test here and be useless. The assertions also pin the shape of the fall: h > 0 (not a total collapse — the fixture only penalizes experts 2 and 3, so a one-hot on 0 or 1 is also a minimum of it), dead == 2 (exactly the two targets), and both survivors above 0.15 (a fall that is really a collapse would fail here and nowhere else).

Two defects this code would otherwise have shipped

Both were caught by the gates above, and both are the kind that read as correct:

1. min_dim(0) for the dead count is a trap. It asks whether every row selected the expert, so with 8 tokens on 4 experts it reads 0.0 everywhere and a perfectly uniform router reports dead=4. max_dim is the right reduce. The comment at the call site says so, because this is the version that looks right.

2. The objective must read probs, not the mask. The mask is built by mask_fill on an argsort index and has no gradient path at all — differentiating it raises backward requires a tracked tensor rather than returning a weak signal. Same reason lb_aux takes the pre-top-k probs, and the same reason its P_i is not the renormalized gates.

Test files that were RED on main

Three MoE test files (11 tests) could not run, before this lane touched anything — proven by git stash on the tree with none of my changes. fdcd030 flipped the schema default use_byteflow to true; these fixtures inherit Default::default(); validate then refuses the combination. Each now pins use_byteflow: false beside the three arms it already pinned — same reason, fourth arm.

---

2. The A/B protocol (deliverable 2) — tools/moe_ab.sh

Ready, not run. The GPU was night2e’s for the whole lane. 6 arms (moe on/off × 3 seeds), 2000 steps, batch 8 × seq 512, pure CE, fp32, --eval-batches 10.

The eval window, stated as a number

10 × 8 × 512 = 40 960 B. This is printed by the script, printed by every eval line, and asserted against the log (assert_window): a run at the wrong window is not comparable to anything, and --batch is a throughput knob no reader would expect to move the scored bytes (§2.6).

Why it re-runs at all

The wave-3 verdict was a tie, and §1.2 says a tie deletes the mechanism — but that tie was read on a stack whose fidelity fix landed after it, so both arms were compared through an instrument that was wrong at the time. A tie measured incorrectly deletes nothing. This is the first honest A/B this arm has had, and it is the reason the script exists rather than a verdict being read off the archive.

Cost is measured, never assumed

§3.3 is explicit that the price of one A/B arm is unknown — AB-PROTOCOL’s 53 min came from ~1.6 s/step on a run whose attention backward never executed, and the 25.8 s/step figure that would replace it has no committed log. So preflight runs 100 steps at the A/B’s own shape and refuses if six arms over BUDGET. There is no hardcoded per-step number anywhere in the file.

Three arm-specific controls, all checked in the log

1. moe= must be present on the routed arm. The utilization field only prints when the routed branch ran; its absence under moe_topk > 0 is the probe::JEPA defect — an arm that cannot show it ran, with a loss curve nobody would question.
2. H and dead are reported, not gated. A collapse is the answer to “does the router learn to specialize”, so the script prints the trace and continues. What is not allowed is reading the BPB without it.
3. The eval line must show engram rows and a non-zero KDA backward — the two §3.2 retractions. Every control on record predates those fixes.

Plus a NaN check and a window check on every arm.

The verdict is against the spread, not against one number

verdict computes the pooled SD of the two arms’ own seed spreads and calls a delta inside it a tie — because §1.2 says a tie deletes. It prints the numbers either way and never decides for a single run.

The protocol’s own checks are gated (tools/moe_ab.sh selftest)

This lane had no GPU, so clean_control, assert_window and the verdict arithmetic were written and never run — which is precisely the defect class this repo keeps finding (§3.2: an eval that read a memory-disabled forward, an attention arm that ran 3126 forwards and 0 backwards, every loss curve healthy). A protocol whose checks have never executed is that shape.

selftest runs 10 cases against synthetic logs: the two clean arms must be accepted, each of the six defects must be rejected, and the verdict must reach both a KEEP and a TIE. It found three real bugs on the day it was written, all of which would have corrupted a real run:

bug	what it did
attention counter read fused kda=F/B	F/B is (fused forward, fused backward) and §3.3 records the fused backward as never running — so it read every clean arm as having no attention backward at all and would have refused all six. The real counter is bwd=.
grep -o 'bwd=[0-9]*'	also matches inside node_bwd=0, so tail -1 returned the custom-node counter. Fixed by anchoring on the leading space.
grep -c ... || echo 0	grep -c prints 0 and exits 1, so the `

The first is the important one: it copied mor_ab.sh’s check, which was correct when the fused backward was the only backward there was, and silently became wrong after §3.3’s fix. A copied gate expires silently.

---

3. The capacity ladder (deliverable 3)

Four rungs, configs/moe-cap-{2,4,8,16}.toml, each small field-for-field with one number changed. small itself is rung 3, so the ladder is 2/3/4/8/16 without duplicating the shipped preset.

Every number below is measured on the instantiated model, not computed from a formula — cargo test -p dormouse-core --test preset_exec -- --ignored, walking the module tree with a ModuleVisitor.

rung	n_experts	total params	compute	memory rows	memory share	param tensors
moe-cap-2	2	8 836 878	5 691 150	3 145 728	35.6 %	48
small (control)	3	9 197 454	6 051 726	3 145 728	34.2 %	54
moe-cap-4	4	9 558 030	6 412 302	3 145 728	32.9 %	60
moe-cap-8	8	11 006 478	7 860 750	3 145 728	28.6 %	84
moe-cap-16	16	13 903 374	10 757 646	3 145 728	22.6 %	132

The closed form is exact at all four rungs

compute(e) = 4 969 998 + e · 360 576 + 1536 · (pad(e) − 8), where 360 576 = 2 · r(d + f + 1) is one TSCT expert’s parameters and pad(e) = 4⌈(3 + e)/4⌉ is the controller’s output width.

rung	predicted	measured
2	5 691 150	5 691 150	exact
4	6 412 302	6 412 302	exact
8	7 860 750	7 860 750	exact
16	10 757 646	10 757 646	exact

This is reported because it is the check, not because the formula is useful. A formula that predicts the instantiated model exactly at four points is one that can be trusted to extrapolate one step to a config nobody has built; a formula that merely agreed approximately would not be.

The one non-obvious term

LinearLike pads out_features to a multiple of 4 (cubek matmul vectorization), and the controller’s output width is 3 + n_experts (three gates plus the router). So the router’s own parameters grow in steps of 4, not one per expert:

e	2	3	4	8	16
pad(e)	8	8	8	12	20
controller params	12 288	12 288	12 288	18 432	30 720

At e = 2, 3, 4 the router costs nothing at all to add — the padding was already there. That is why the ladder must not be read as “each expert costs the same”: the per-expert cost is 360 576 (the TSCT factors) and the router is a step function of it, and the step function is invisible below e = 5.

FLOPs per token per iteration, small geometry

d = 768, f = 2048, r = 64, max_iter = 4, 4096 tokens/step.

	per expert	×4 iterations	note
TSCT (shipped)	4·r·(d+f) = 7.2 MFLOP	28.9 MFLOP/token	two matmuls at rank 64
dense (no TSCT)	4·d·f = 6.3 MFLOP	25.2 MFLOP/token

TSCT is 8.75× CHEAPER in FLOPs than the dense FFN it replaces, because r = 64 against d = 768. This is the strongest thing the ladder says and it is not a MoE result: the win is the spectral factorization, and it applies at e = 1 too. What the expert count adds on top of it is capacity, not speed.

What the ladder does not say

• It does not say more experts is better. No rung has run the A/B. Capacity without a verdict is a price list.
• It does not include the KDA arm or the embedding, so these are expert-bank prices, not whole-model step costs. A step is launch-bound (AGENTS.md §3.1: mean GPU utilization 13.3 %, 79 % of samples ≤ 5 %), and FLOPs are not where the win is at this width.
• The rung files are deliberately in the --ignored cost cell. A full-width small instantiation measures ~195 s on this box’s CPU backend (measured: 131 s / 258 s / 506 s / 1016 s for the four rungs); four more in the default cell would put ~13 min into every cargo test.

---

4. TSCT at 50M–1B (deliverable 4) — notes, not implementation

Two costs scale with the model and neither amortizes over the batch.

4.1 The retraction is a fixed per-step cost, and it is launch-bound

polar_orthogonalize(x, iters) on a factor canonically shaped [k, m] (k = r):

step	work	FLOPs
Gram XXᵀ	[m,k]·[k,m]	2k²m
5 power iterations	[k,k]·[k,1] each	negligible
Rayleigh quotient	2 more [k,k]·[k,1]	negligible
NS, per iteration	XXᵀ + (XXᵀ)² + ·X	4k²m + 2k³

At iters = 3: 14k²m + 6k³ FLOPs per factor.

At small’s geometry (k = 64, m = 768, 14 factor tensors: U and V of each of the 7 TSCT linears) that is ≈ 638 MFLOP of arithmetic per step against a measured 52–61 ms. At the box’s ~8 TFLOP/s fp32 ceiling that arithmetic is ~0.08 ms — so the retraction is >99.8 % launch overhead.

The consequence is the important one: the retraction’s cost tracks the NUMBER of factor tensors, not the FLOPs. Each factor is its own sequence of ~40 tiny launches on a [64, 768] matrix, and no amount of extra width makes any one of them cheaper.

At 1B this does not get better by itself:

• n_experts up → factor count up linearly. The ladder prices parameters; retraction time follows the same line.
• d_model up → m up → FLOPs up as k²m, but launches unchanged. So the arithmetic share rises with width, and at some width the retraction stops being pure overhead. Where that crossover is has not been measured and it is the number that decides whether a 1B run needs a batched retraction.
• Measured already, and unchanged by any of this: 8× the data costs 1.2× the retraction time (AGENTS.md §3.1). --retract-every 1000 gives retr=0.0 and a 188 ms step against 240 ms.

The mitigation already in the tree is --retract-every (cadence), and there is a grouped arm (retract_batched, RETRACT_BATCHED) that fuses same-shaped factors into one [B, m, k] call. At 1B the grouped arm should be the default, not an opt-in, because its whole benefit is amortizing exactly the per-factor launch count identified above. This is not implemented and is not claimed.

4.2 The max_ortho latch is likelier to fire at scale, and the failure is silent by design

The metric is per-entry ‖UᵀU − I‖_F / rank; the threshold is 1e-3 one-way; above it all factors irreversibly switch to fp32, checks stop, and the latch persists in the checkpoint.

Why scale makes it worse, and this is arithmetic rather than opinion:

• The threshold is an absolute per-entry number against a quantity whose natural scale is 1/√m — a perfect orthonormal [m, k] factor reads ≈ 1/√m per entry (3.6e-2 at m = 768, 2.2e-2 at m = 2048). The margin between “fresh init” (1.4e-4) and the threshold is comfortable at m = 768 and is a ratio against a quantity that shrinks as m grows.
• A 1B model needs more factors, so the worst-case-over-factors statistic that max_ortho reports has more draws from the same distribution. With n_experts × 2 + 2 factors instead of 14, the maximum of that sample is higher at any fixed drift rate — the classic max-of-N tail, and the latch is a threshold on exactly that maximum.
• The quantization is what moves it: fp8/fp4 factor quantization breaks orthogonality, and the whole point of the latch is to catch that before the forward silently degrades.

The failure mode is silent on purpose and that is the risk. The latch’s own design — one-way, checks stop, fp32 forever — means a run that latches at step 300 and a run that never latches produce the same loss curve shape, differing only in speed, with the latch state in the checkpoint and on one eval line (tsct=). A 1B run that spends its whole budget in fp32 factors has silently given up the weight-quantization ladder (§6 of tsct-spec) without any loss number changing.

So at 1B the latch needs to be watched as a first-class metric, not as a rare event, and the cheap instrument is already built: max_ortho is on the eval line. What is missing is a threshold alert rather than a print — and the per-entry-vs-1/√m ratio above is the reason a single absolute 1e-3 is the wrong shape at scale.

4.3 What I would measure first at 50M, in order

1. max_ortho per factor, not worst-case, for 500 steps — answers 4.2 without any new machinery (TsctDiag already walks factors).
2. retr ms vs --retract-every {1, 5, 10} at batch 8 and 32 — the cadence question, and §3.3 records it as never measured.
3. retract_batched vs retract_factor at 16+ experts — is the grouped arm’s win growing with the factor count, as the launch-bound reading predicts?

---

5. Follow-ups (not done, not in this lane)

1. forward_full_state’s loss_mask is not plumbed through the byteflow path. 98f98d7 fixed the merge collision that left main uncompilable and passed an honest None there, because forward_channel takes only (input_ids, targets) and has no mask to forward. An SFT run with use_byteflow would score every position and ignore the mask. Closing it is a parameter through forward_channel and its three callers — the SFT lane’s file. Named, not fixed.
2. benches/history.tsv has an unresolved merge conflict on main at lines 163–167 (<<<<<<< HEAD / >>>>>>> origin/wt/adjoint-tune), between two rows both of which claim 2026-10-03. Left alone — it is two lanes’ row and resolving it is not mine to guess. This lane’s row is appended after it.
3. lb_aux is computed every routed iteration even when moe_lb_coef == 0, which is the shipped default. That is 3 wasted device reductions per step on the default path. Deleting it would remove the arm’s own removal control, so it is a config question rather than a free win.
4. top-k still does not dispatch (tsct-spec §12). Everything in §3 is a capacity price list; the compute saving is still §12’s priority 1 and is untouched by this lane.
5. moe-cap-16 has never been instantiated on the GPU. Its parameter count is measured (§3); its VRAM footprint at batch 8 is not, and at e = 16 the expert bank is 132 tensors where small has 54.

---

6. Reproducing

# gates (CPU, ~2 min for the core lib)
cargo test -p dormouse-core --lib moe::
cargo test -p dormouse-core --test moe_routing_seam --test moe_grad_seam --test moe_step1_gate

# the capacity ladder's execution gates (~1 s; the COST gate is --ignored, ~32 min)
cargo test -p dormouse-core --test preset_exec -- moe_cap the_capacity_ladder every_shipped_preset

# the A/B, once the card is free (§1.5: this waits, it does not interrupt)
tools/moe_ab.sh selftest   # 10 cases, no GPU needed - run this first
tools/moe_ab.sh preflight  # prices one arm before committing hours
tools/moe_ab.sh all
tools/moe_ab.sh verdict