Skip to content

TSCT diagnostics + the 1.05 comment — 2026-10-01

Lane: the external review’s pre-100k checklist, items 1 and 2 (docs/research/2026-10-01-alphaxiv-answers-batch-3.md, “ПРЕДЗАПУСКОВОЙ ЧЕКЛИСТ”). Gates tonight’s 100k launch.

Three rare diagnostic series on the eval line, plus the comment fix. Cadence: eval steps only (default every 500). Every number is a host-side read of already-materialised factors at a boundary the run already syncs on — no new device sync was introduced (§1.3).

(a) Gram error of the FP masters, BEFORE and AFTER retraction

Section titled “(a) Gram error of the FP masters, BEFORE and AFTER retraction”

No new metric. LinearLike::max_ortho already is the per-entry masters reading (‖UᵀU−I‖_F / rank, AGENTS §2.3), so the pair is that function called either side of retract_tsct* in train_loop. Ladder rung 2: it already existed in this codebase.

(b) Gram error of the EFFECTIVE FORWARD factors at the current alpha

Section titled “(b) Gram error of the EFFECTIVE FORWARD factors at the current alpha”

burn_spectral::ortho_error_forward(u, alpha) = ortho_error_per_entry(ste_ternary_annealed(u, alpha)) — the same function SpectralLinear::forward calls, at the layer’s own alpha. Computed on the ternary-shaped effective factor, never on the raw master, which is precisely the metric that would look perfect while the forward degrades.

(c) The |s| spectrum: max, min, count of near-off ranks

Section titled “(c) The |s| spectrum: max, min, count of near-off ranks”

burn_spectral::spectrum_stats(s, 1e-3) → (max, min, off). off counts |s_i| < 1e-3 (“почти выключенные ранги”), summed across every factor in the model. 1e-3 is ~3 orders below the per-entry magnitude of a [768,64] orthonormal factor (~3.6e-2); it is a reporting threshold, nothing branches on it.

On a freshly built [768,64] TSCT factor (fp32, CPU, rank 64 — the trainer’s shape), measured with the new functions:

metric value
masters on the manifold (‖UᵀU−I‖_F/64) 2.27e-7
forward factor at alpha = 0 2.27e-7 (identical — the cross-check)
forward factor at alpha = 0.25 1.64e-2
forward factor at alpha = 0.5 3.08e-2
forward factor at alpha = 0.75 4.33e-2
forward factor at alpha = 1 (the shipped default) 5.39e-2

And on a real SpectralLinear::new(768, 768, 64) layer:

metric value
masters 8.06e-8
forward at alpha = 1 7.96e-2
forward at alpha = 0 8.06e-8
|s| at init max 1.0, min 1.0, off 0

The gap is ~10⁶ and it is the reviewer’s warning, confirmed. The masters are on the manifold to fp32 precision; the factor the forward actually multiplies by is 5-8 orders of magnitude further from it. The ternary projection with its 0.7·mean dead zone is not an orthogonal map, and max_ortho — the only metric the trainer had — cannot see that by construction: it is a function of the masters alone.

Two consequences that are facts about tonight’s run, not forecasts:

  1. The trainer’s max_ortho 1e-3 latch cannot see this. It reads 8e-8 on a factor whose forward factor is at 8e-2, so the latch never fires and the degradation it was installed to catch is invisible to it. The latch’s threshold is not wrong for what it measures (master drift) — it is just not a forward-health metric, and nothing in the log claimed it was. The live 2-step run above is the same thing at step 1: aft = 2.12e-8 (latch quiet) against fwd = 6.59e-2 (66× over the latch’s own threshold).
  2. alpha is 1.0 everywhere and nothing in the repo sets it (grep set_alpha → only burn-spectral’s own SpectralLinear::forward and its diag example). So today’s 100k run is in the alpha = 1 column: the worst one, and the column the metric now reports. Whether that 8e-2 per-entry Gram error hurts BPB is exactly what the pre-100k A/B (checklist item 3) is for; this lane measures it, it does not decide it.
tsct=<bef>/<aft>/<fwd>/<smin>/<smax>/<off>@a=<alpha>
  • bef/aft — masters’ per-entry Gram error either side of THIS step’s retraction. - when retract_every did not fire on the eval step (two identical numbers printed as a pair would read as a measurement and mean “we did not measure it” — the same reasoning that put names into the kda_asked destructuring).
  • fwd — forward-factor Gram error, worst over factors, at the layer’s alpha.
  • smin:smax — the |s| spectrum’s range over every factor.
  • off — near-off rank count, summed.
  • a= — the largest alpha folded in. fwd is a function of alpha, so a number without it is not comparable across runs.
  • max/sum conventions match max_ortho (worst-case) except s_min, which is a min because a min is what “a rank died” looks like, and off, which is a sum because one dead rank in one factor is the fact being reported.

None (no field at all) when the model has no TSCT factors — a --set use_tsct=false run prints nothing rather than a row of zeros.

A REAL eval line (2-batch CPU run, --preset nano, 2 steps)

Section titled “A REAL eval line (2-batch CPU run, --preset nano, 2 steps)”
step 1 EVAL ce=5.423 bpb=7.824 BEST over 256 B (fixed window) fused kda=0/0 asked=0 bwd=0 declined=0 ops=0 node_bwd=0 norm=0/0 muon_skipped=0/0 engram=8/8 fb=0/0 tsct=6.90e-6/2.12e-8/6.59e-2/1.000e0/1.000e0/0@a=1.00

Read the tsct= field left to right:

slot value what it says
bef 6.90e-6 masters’ per-entry Gram error BEFORE this step’s retraction
aft 2.12e-8 …and AFTER: the retraction does its job, 325× better
fwd 6.59e-2 the factor the forward actually multiplies by
smin:smax 1.000e0 : 1.000e0 s is all-ones at step 1, as initialised
off 0 no dead ranks yet
a 1.00 the annealing alpha the fwd number is at

This is the whole finding in one line. The retraction is doing its job (aft is 325× better than bef) and the masters are then held at 2.12e-8, far under the 1e-3 latch — while the factor the forward multiplies by reads 6.59e-2, which is 66× OVER that same latch’s threshold. The latch cannot fire: it is a function of bef/aft alone, and those are the two healthy numbers. Nothing in the log before this field could have told a reader that the deployed factor is 66× past a limit the model is nominally inside.

burn-spectral’s POWER_ITERS doc and both mul_scalar(1.05) sites claimed 1.05 was a safety factor covering the power-iteration error, and the old comment justified it as “converges from below, so the true sigma_max stays inside the basin”. False as measured (2026-09-30, AGENTS §3.7 tail): over 200 random [768,64] factors the estimate is worse than 5 % on 32–47 % of draws and the prescaled sigma_max reached 1.087.

Rewritten to what is true, with the arithmetic:

  1. Cubic suppression. Expanding about s = 1, p(1+e) − 1 = (5/2)·e³ + (15/8)·e⁴ + (3/8)·e⁵ — linear and quadratic terms cancel exactly (p′(1) = 0), so error is damped cubically. This is why the mis-estimate is benign: measured prescaled values up to 1.087 still return σ = 1.000000000.
  2. The basin, which is the real property. p(s) − s = 0 at s² = 1 and s² = 7/3, sign negative between, so the attracting interval is 0 < x₀ < √(7/3) ≈ 1.5275. A 1.05 prescale therefore tolerates an estimate as bad as 1.5275/1.05 = 0.687 — a 31 % underestimate — i.e. 6× the error the factor was documented as covering.
  3. POWER_ITERS bounds the estimate, not the factor; it is p′(1) = 0 that absorbs the residue.

The literal 1.05 is now const SIGMA_OVERSHOOT (three call sites), so the comment and the arithmetic cannot drift apart again.

Why c* is not learned (stated in the comment, because “just make it learnable” is the obvious wrong next move): c multiplies the whole factor, so a gradient on it is a gain-control mechanism — the optimizer would shrink the factor to cut the loss, which is the exact thing the retraction exists to prevent. c* = argmin_c max_{x∈[ℓ,u]} |p³(x/c) − 1| is measurable offline from saved factors if the margin ever stops being enough.

Tests (all red→green by perturbation, not by literal)

Section titled “Tests (all red→green by perturbation, not by literal)”

burn-spectral (cargo test -p burn-spectral --lib, ndarray):

test what makes it red without the metric
ortho_per_entry_reports_a_known_perturbation fixture [768,64] with diag(1+eps) on the first k rows ⇒ Gram error exactly ((1+eps)²−1)/√k in closed form. eps = 0 must read < 1e-7; eps = 0.05 must read 1.281248e-2 vs closed form 1.281248e-2. An unperturbed read cannot produce it.
ortho_per_entry_is_zero_on_the_manifold a retraction applied to an already-orthonormal factor must not report drift
ortho_forward_departures_from_the_masters_as_alpha_rises three properties: at alpha = 0 exactly equals the masters’ metric to 1e-12; at alpha = 1 is > 100× it; monotone in between. Plus assert_eq! that the masters’ metric returns the SAME number for any alpha — that is the defect this series exists to expose.
spectrum_stats_reports_max_min_and_dead_ranks fixture (2.0, 1e-4, −0.5, 0.0, 1.25) ⇒ (2.0, 0.0, 2); tolerance moves only the count (1 / 2 / 3 at 1e-4 / 1e-3 / 1.0); an all-alive spectrum reads 0, so a constant-returning metric fails

Why the perturbation fixture is built rather than applied to a real factor: perturbing an existing orthonormal U by ε moves the off-diagonal Gram entries by ε·U_ij, so the expected value would depend on the fixture’s own cross terms instead of on the measurement. The diag(1+eps) fixture has none, which is what makes the expected number a property of the METRIC.

dormouse-train (cargo test -p dormouse-core -p dormouse-data -p dormouse-train --lib):

test what it pins
tsct_field_format_is_golden_and_alpha_zero_is_the_cross_check the exact tsct= string; a dense linear prints no field; at alpha = 0 the forward metric equals max_ortho to 1e-12 relative (the built-in cross-check); at alpha = 1 they differ by > 100×; missing sides render -
tsct_diag_runs_on_a_real_two_batch_cpu_eval the walk survives the real train_loop eval path (Run harness: batch 2, eval_batches 2, eval_every 1) over every factor incl. the ones lm_head’s fold owns — a unit test cannot see a fold that forgot lm_head
cfg_retract_every_default_is_one the bef/aft pair is fully populated only because retract_every is 1. The external review asks for every-4th: if that moves, this assertion fires and the condition to re-check is eval_every % retract_every == 0

Carried, not fixed (other lanes / owner decisions)

Section titled “Carried, not fixed (other lanes / owner decisions)”
  • max_ortho’s 1e-3 latch is not a forward-health metric, per the measurement above. Changing the latch to fire on fwd is a POLICY change with a one-way fallback attached and belongs to the owner, not this lane. The field now reports both so the decision can be made from a log.
  • alpha has no schedule. Nothing in the repo sets it; the 100k run is at alpha = 1. Annealing alpha is checklist item 3’s business.
  • A train at alpha = 1 runs the 8e-2 forward factor today. That is a measurement, not a verdict: whether it costs BPB is the A/B.
  • Under a non-Fp32 --quant (the sm_120 auto default is Fp8), fwd reports the TERNARY arm, not the deployed one — LinearLike::forward takes forward_quant (quant_factor(master)) whenever quant != Fp32, and the ternary factor is not what it multiplies. Named here rather than silently averaged in. Follow-up if it matters: fold quant_format into the walk and report both.
  • Checklist item 3 (intervals 4 / 16 / none, ≥2 seeds, BPB on equal bytes AND equal wall-clock) and item 4 are untouched — they are runs, not code.

Worktree wt/tsct-diag, branch off 3e1812b. /home was at 98 % with 12 G free — a cold worktree build is ~41 GB, so this lane reused the shared warm target/ dirs (CARGO_TARGET_DIR=…/target and …/vendor/dormouse-fused/target) through tools/build_lock.sh. All cargo went through the build lock. No cleanup was needed: the lane created zero bytes of target dir.