Capability per byte and per GB for small LMs: research survey (2024-2026)

Scope: what the 2024-2026 literature says about getting the most capability out of a byte-level LM that must train and live on one 16 GB consumer GPU (5060 Ti) with 64 GB host RAM. Every claim carries an arXiv ID or URL. Section notes in italics tie findings back to dormouse’s current design.

Survey date: 2026-09-21.

1. Small-LM architecture SOTA

Hybrid linear:full attention ratios converge near 3:1. Three independent 2025 results agree on the ratio, not just the idea. Samba (Mamba + sliding-window attention, 3.8B / 3.2T tokens) beats pure attention and pure SSM baselines and extrapolates 4K→256K (arXiv:2406.07522). Kimi Linear interleaves KDA with full attention at 3:1 and, under an identical recipe, beats full MLA on short-context, long-context, and RL regimes, with ~1.16x compute efficiency in scaling-law fits (arXiv:2510.26692). Qwen3-Next uses Gated DeltaNet:gated attention at 3:1 plus an ultra-sparse MoE (512 experts, 10 routed + 1 shared) and claims Qwen3-32B-class quality for under 1/10 the training cost (Qwen blog: https://qwen.ai/blog?id=4074cca80393150c248e508aa62983f9cb7d27cd). The consistent caveat: some full (or sliding) attention layers are required for precise recall; pure linear models lose it (arXiv:2406.07522, arXiv:2510.26692).

KDA details that matter. Kimi Delta Attention extends Gated DeltaNet with channel-wise (per-dimension) decay instead of head-wise, a Diagonal-Plus-Low-Rank chunkwise formulation, and gets positional information from KDA itself: their attention layers run NoPE without losing long-context quality (arXiv:2510.26692). Gated DeltaNet-2 goes further and decouples erase vs write gates, reducing to KDA when gates collapse (arXiv:2605.22791).

MoE at small scale: fine-grained, token-choice, no shared expert. OLMoE (1.3B active / 6.9B total, 5.1T tokens) ran the controlled sweeps: 64 small experts with 8 routed beats 8 experts with 8 routed; token-choice routing beats expert-choice at matched budget; adding a shared expert did not help and can hurt by reducing routing flexibility. MoE matched dense 7B quality with under half the training FLOPs (arXiv:2409.02060). MoE therefore works at 1B-active scale, but note OLMoE pays for it in total-parameter memory, which is exactly the resource dormouse is short on; the Qwen3-Next result shows the loss benefit of more total experts at fixed active count (Qwen blog above).

Weight-shared recurrent depth has real evidence at small scale. Mixture-of-Recursions combines a shared layer stack with per-token depth routing; across 135M-1.7B it sets a Pareto frontier: lower validation perplexity and better few-shot accuracy at equal training FLOPs, plus higher throughput. Middle-cycle parameter sharing was the best sharing layout (arXiv:2507.10524). A 3.5B recurrent-depth model improves sharply with test-time iterations (ARC ~49% at r=4 → ~70% at r=32, 0.8T tokens) (arXiv:2502.05171). Energy-Based Transformers report ~35% faster scaling w.r.t. data/depth/params/FLOPs and +29% more gain from extra inference compute than Transformer++ (arXiv:2507.02092).

PonderNet-style halting: evidence is weak, the loop is not. The original PonderNet validated on algorithmic and bAbI-scale tasks only (arXiv:2107.05407). A 2026 audit of a PonderNet-style halting reasoner found the apparent advantage of learned halting annihilates to +0.000 across 31 configurations once the readout is equalized; the value lies in the recurrent refinement, not the halting head (arXiv:2608.22347). MoR replaced halting with lightweight per-token depth routers and won (arXiv:2507.10524).

Dormouse: the 3:1 linear:full ratio matches the LoopBlock’s router blend; keep at least one spatial-attention-heavy iteration per 4. The loop itself is well supported (MoR, Geiping, EBT). The PonderNet halt head is the weak link, section “What to DELETE”.

2. Tokenizer-free / byte-level

Byte models lose at small compute, close the gap at scale, and win on robustness. ByT5 showed byte models need more compute to match subword quality but are more robust (arXiv:2105.13626). bGPT (110M) matched specialized models across text/audio/image and simulated CPU state from bytes (arXiv:2402.19155). BLT is the scaling landmark: entropy-patched byte models match Llama-3-style BPE models up to 8B params / 4T bytes with better scaling at fixed inference cost and better long-tail robustness (arXiv:2412.09871, ACL 2025).

Patching strategy decides everything at small scale. SpaceByte, a byte transformer with larger “global” blocks applied after space-like bytes, roughly matches a SentencePiece transformer at fixed training and inference compute: PG-19 1.009 vs 0.989 BPB, arXiv 0.748 vs 0.768 (byte wins), Github 0.500 vs 0.508 (byte wins); fixed-size patching materially degrades it, so the dynamic boundary rule carries the gain (arXiv:2404.14408). H-Net learns chunking end-to-end; one stage matches BPE only at >1B params, two stages match a token model of 2x size, and the byte advantage grows on code, Chinese, and DNA (~4x data efficiency) (arXiv:2507.07955). MegaByte fixed 4-byte patches started this line and is the floor, not the ceiling (arXiv:2305.07185, cited in both above).

Why subwords win is now decomposed, and dormouse can buy each ingredient. A 2026 controlled study on a 1.7B byte LM isolates the benefits subword tokenization actually delivers: (1) sequence compression, i.e. more bytes of context per step, (2) morphological boundary priors, (3) a larger input vocabulary. They replicated each effect on the byte model with multi-head n-gram input tables (~70M params, ~35k-vocab equivalent) and boundary signals and observed significant pretraining improvements per effect (arXiv:2604.27263). Bolmo shows a different path: byteify an existing subword LM for under 1% of a pretraining budget and reach competitive 1B/7B byte LMs (arXiv:2512.15586).

Dormouse: at 48M-row Engram plus small steps, dormouse is effectively buying ingredient (3) and part of (1) already. The cheapest documented win is ingredient (1) done explicitly: space- or entropy-patched higher-throughput steps for the transformer trunk, byte-level prediction unchanged.

3. N-gram / lookup memory

Off-accelerator lookup tables are now a published, winning recipe. SCONE adds frequent-n-gram embeddings computed by a separate model, stored off-accelerator after training, and queried with O(1) lookups; a 1B accelerator-resident model beats a 1.9B dense baseline across corpora at roughly half the inference FLOPs and accelerator memory, and loss drops monotonically as n-gram embeddings scale (arXiv:2502.01637, NeurIPS 2025). DeepSeek’s Engram paper frames the same module, modern hashed n-gram memory with O(1) lookup, as a third sparsity axis beside MoE and sparse attention (arXiv:2601.07372). This is the exact pattern dormouse’s host-RAM Engram implements, including the report’s “n-gram tables with Adam, no weight decay” rule.

Lookup memory beats FFN capacity for facts at equal compute. Meta’s Memory Layers at Scale replaces FFNs with trainable sparse key-value layers; at up to 128B memory params / 1T tokens, memory-augmented models beat dense models with over 2x the compute and beat MoE at matched compute and params, with the largest gains on factual QA (arXiv:2412.09764, ICML 2025).

Table scaling laws on the input side are log-linear; the output side is a trap. Over-Tokenized Transformer finds a log-linear relation between input vocabulary (hierarchical n-gram tables) and training loss, holds across model sizes, and shows a large input table alone recovers a 2x-size baseline; scaling the output vocabulary, by contrast, hurts smaller models (arXiv:2501.16975, ICML 2025). BLT’s byte n-gram ablation found hash embeddings beat frequency-based ones (arXiv:2412.09871 appendix J). Hash collisions are measurable interference: TN-gram replaces Engram-style per-order hash tables with a tensorized CP decomposition and reports gains (arXiv:2606.08347).

Neural + n-gram interpolation still helps big models. Infini-gram indexes 5T tokens with suffix arrays and shows ∞-gram interpolation improves neural LMs up to 70B, evidence that count statistics carry signal neural nets leave on the table at every scale (arXiv:2401.17377). N-Grammer added latent n-gram tables to transformers on C4 with gains over Primer-class baselines (arXiv:2207.06366).

On the Adam-without-weight-decay rule: public ablations isolating weight decay on hash-lookup tables are thin. What is public: Meta’s memory layers scale with Adam-family updates and the optimizer-state memory of huge value tables is a first-class engineering cost (arXiv:2412.09764); Moonlight’s routing puts embeddings/heads on AdamW while matrices get Muon with weight decay (arXiv:2502.16982). The no-WD-on-tables rule in the Qwen3.8-Flash-Next report is plausible but not independently replicated; keep it, it costs nothing, but do not defend it as settled.

Dormouse: the 48M-row / 1.5B-param Engram is the best-evidenced component in the repo (SCONE + Engram + Over-Tokenized + Memory Layers all agree). Two upgrades are supported: (a) grow input-side tables, they pay log-linearly, while never adding output-side tables; (b) consider cross-order shared factors or lower collision load if A/B shows Engram is collision-bound (arXiv:2606.08347).

4. Data engineering for small compute

This is the largest documented lever at fixed compute, and it transfers down-scale. DCLM: model-based filtering produced a 7B model at 64% MMLU 5-shot on 2.6T tokens, +6.6pp MMLU over the previous open-data SOTA with 40% less compute; 412M-parameter probes still predict which datasets win at 7B (arXiv:2406.11794). FineWeb-Edu: an LLM-annotated educational classifier lifted a 1.71B model trained on 350B tokens from 33% to 37% MMLU and 46% to 57% ARC over FineWeb, roughly +2% aggregate (arXiv:2406.17557). SmolLM2 confirmed the same ordering below 1B: DCLM filtered with the edu classifier at threshold ≥3 beat both parents for 135M/360M models (arXiv:2502.02737).

Small-scale decision procedure is solved. DataDecide (25 corpora, 4M-1B, 3 seeds): ranking models at 150M predicts the 1B winner ~80% of the time; no scaling-law method beat single-scale ranking; character-normalized likelihood metrics make MMLU/ARC/HellaSwag/MBPP/HumanEval >80% predictable with 0.01% of target compute (arXiv:2504.11393, ICML 2025). RegMix: 512 runs of 1M models on 1B tokens suffice to fit a mixture regressor whose pick beats human and DoReMi selections for a 1B model at 10% of the compute; mixture choice swings single-task performance up to 14.6% (arXiv:2407.01492, ICLR 2025). Ultra-FineWeb adds a cheap loss-based verification loop for filter candidates (arXiv:2505.05427).

Repetition budget for a 44 GB corpus. Up to 4 epochs of repeated data costs almost nothing (7B model, 4 epochs: +0.5% validation loss vs 1 epoch on 4x unique data); value decays to zero by ~16-40 epochs, and under repetition the optimal allocation shifts compute toward more epochs rather than more parameters (arXiv:2305.16264).

Dormouse: the corpus is ~44 GB of raw UTF-8, single file, no documented dedup or quality filtering. At dormouse scale the DCLM/FineWeb-Edu deltas are the difference documented as +12% relative MMLU (1.7B) and +6.6pp (7B) for filtering alone. Two concrete plays: filter or replace the corpus with an edu-class-filtered corpus re-serialized to bytes, and run a DataDecide-style BPB ranking at small preset before any long run.

5. Objectives

Multi-token prediction: real but size-dependent; byte-level version looks strong. Gloeckle et al.: n independent future-token heads, no training-time overhead; 13B models solve +12% HumanEval / +17% MBPP; gains grow with model size, so at <1B expect small effects; notably, in their byte-level experiment, 8-byte prediction solved 67% more MBPP pass@1 than next-byte (arXiv:2404.19737). DeepSeek-V3 ships one-depth MTP (arXiv:2412.19437), Qwen3-Next trains with MTP and reports both a pretraining-quality and an inference-speed payoff (Qwen blog above). DSpark at K=4 is in family with this evidence.

Distillation beats supervised pretraining for students when a teacher exists. Distillation Scaling Laws (143M-12.6B students): with an existing teacher, distillation dominates supervised training up to a predictable compute level; if the teacher must also be trained for a single student, supervised wins; a too-strong teacher eventually hurts the student (capacity gap) (arXiv:2502.08606, ICML 2025). The optimal teacher sits near 2.5x the student scale (arXiv:2311.07052). Method: on-policy GKD with reverse KL/JSD on student rollouts (arXiv:2306.13649, arXiv:2306.08543); the Thinking Machines writeup shows a Qwen3-class base model trained to RL-levels at a fraction of RL compute with per-token reverse KL on student trajectories (https://thinkingmachines.ai/blog/on-policy-distillation).

JEPA-style latent objectives at LM scale: first positive results, pretraining evidence thin. LLM-JEPA (NeurIPS 2025) adds an EMA-free embedding-prediction loss to LLM training: Llama-3.2-1B-Instruct +14.2pp on NL-RX-SYNTH fine-tuning, 5-25pp gains across Gemma-2/OpenELM/OLMo, GSM8K gains on Qwen3-1.7B and R1-distill-1.5B, and reports overfitting robustness; pretraining experiments exist but are small-scale (arXiv:2509.14252). EBT’s contrastive energy head over (context, candidate) pairs is the InfoNCE-shaped version and carries the strongest scaling evidence (arXiv:2507.02092).

KoLeo at LM scale: no public evidence. KoLeo comes from DINOv2 vision training (arXiv:2304.07193); no 2024-2026 LM paper shows it pulls weight on top of CE.

Dormouse: JEPA aux is directionally supported but its gains are documented in fine-tuning; at 0.05 weight it is cheap insurance against collapse, keep. The evidence-backed addition is distillation: dormouse is small, teachers are free to download, and reverse-KL on-policy is a training-loop change, not an architecture change.

6. Optimizers and stability

Muon is the documented compute winner. Moonshot’s scaling-law study: Muon reaches AdamW-equal loss with ~52% of the training FLOPs under compute-optimal settings (fitted loss 2.506·C^-0.052 vs 2.608·C^-0.054); two ingredients made it scale: weight decay on matrix params, and matching Muon’s update RMS to AdamW’s so AdamW-tuned LR/WD transfer directly (γ ← 0.2·max(A,B) adjustment) (arXiv:2502.16982). Routing rule of record: Muon on 2D linear maps, AdamW on embeddings, heads, norms, scalars (arXiv:2502.16982, arXiv:2412.09764 practice).

An Adam-side upgrade exists for the non-Muon groups. AdEMAMix (two mixed EMAs): a 1.3B LM trained on 101B tokens matches AdamW on 197B tokens, +95% (arXiv:2409.03137, ICLR 2025). Relevant to dormouse’s host-RAM CPU-Adam tables: the second EMA is one extra accumulator per row.

Precision: the low-bit window is real but has a floor. Precision-aware scaling laws (465 runs, 3-16 bit): lower training precision lowers effective parameter count with saturating returns around 6-8 bits; both 16-bit training and sub-4-bit training are suboptimal; models trained in lower precision degrade less under post-training quantization (robustification dominates) (arXiv:2411.04330). An FP-specific follow-up finds exponent bits matter slightly more than mantissa bits and a critical data size beyond which more tokens in low precision actively hurt (Sun et al., ICML 2025, https://proceedings.mlr.press/v267/sun25j.html).

Dormouse: Muon+ with RMS matching and the 2D-only policy is exactly the validated recipe; keep. The stability stress protocol (constant-LR spike counting) matches how Moonlight validated stability. One flag: the fp8 factor-forward interacts with the documented overfit-NaN episodes on this box, and the precision laws predict fp8-range problems grow as the model overtrains the small test corpus; --quant fp32 remains the safe arm.

7. Param-efficiency tricks that change training decisions

Layer sharing pays at sub-billion scale. MobileLLM: deep-thin architectures plus embedding sharing plus GQA give +2.7%/+4.3% over prior SOTA at 125M/350M; immediate block-wise weight sharing adds +0.7%/+0.8% more at near-zero latency cost (arXiv:2402.14905, ICML 2024). Relaxed Recursive Transformers convert existing models to shared-layer recursion with layer-wise LoRA restoration at minimal quality loss (arXiv:2410.20672). MoR’s middle-cycle sharing is the pretrained-from-scratch version (arXiv:2507.10524). Geiping’s 3.5B loop confirms shared-depth scales past 1B (arXiv:2502.05171).

Weight tying is a real but small-model-sized decision. Tying input embeddings with the output head improves small-LM perplexity (arXiv:1608.05859) and is standard for sub-billion models (arXiv:2402.14905). At a 256-byte vocabulary the head is only 256·d params, so the memory argument vanishes either way; decide by loss, not bytes.

Low-rank expert bodies: no direct public pretraining win. Fine-grained MoE gains are documented with plain small FFN experts (arXiv:2409.02060). TSCT-style low-rank/spectral expert factors have no 2024-2026 public comparison at matched active params against dense small experts; the closest published lines are LoRA-expert MoE variants and the spectral analysis in Muon’s orthogonality story (arXiv:2502.16982). Treat TSCT as an unproven-swap, A/B against plain fine-grained SwiGLU experts at matched params before trusting it.

GQA / head reuse is free capacity. Grouped-query attention is weight sharing across KV heads and is now default in sub-billion recipes (arXiv:2402.14905).

Dormouse: the LoopBlock already is the strongest param-efficiency play in the file (section 1). Untied lm_head is fine at vocab 256. The one open empirical question is TSCT vs plain small experts.

Top 10 highest-leverage moves for dormouse

#	Move	Expected BPB/loss gain	Implementation cost	Risk	Evidence strength
1	Quality-filter the corpus (edu-classifier style), then dedup; re-eval on the 30 MB held-out tail	Largest single lever: +12% rel. MMLU at 1.7B, +6.6pp MMLU at 7B for filtering alone (arXiv:2406.17557, arXiv:2406.11794); BPB impact on raw text is smaller than MMLU impact but positive	Medium: run a small classifier over 44 GB, re-serialize	Low; filtering too aggressively shrinks unique bytes	Strong
2	Adopt a DataDecide/RegMix protocol: 20-40M-param preset ranking of data/config candidates on 100-200M bytes, decide by BPB	Not a direct gain; it is the multiplier that makes every other row cheap and keeps you from shipping regressions	Low: the trainer already supports small presets	Low	Strong (arXiv:2504.11393, arXiv:2407.01492)
3	Space- or entropy-patched patching for the trunk: transformer steps over ~4-8-byte patches, byte-level heads unchanged (BLT/SpaceByte-style), or precompute entropy from the current model	BPB: SpaceByte matched SentencePiece at fixed compute; BLT shows BPB-per-FLOP scaling improves with patching; at dormouse size expect closing most of the byte-vs-BPE gap (arXiv:2404.14408, arXiv:2412.09871, arXiv:2604.27263)	Medium-high: data pipeline + masking changes, respects the no-4D-slicing rule	Medium: checkpoint-format churn; entropy patching needs a helper byte LM	Strong for the gap, medium for exact magnitude at 50M-scale
4	Keep and grow input-side Engram tables; never add output-side lookup; monitor collision load	SCONE: monotone loss drop with table growth, 1B(+tables) > 1.9B dense; Over-Tokenized: log-linear input-vocab→loss, output-side hurts (arXiv:2502.01637, arXiv:2501.16975)	Low: --engram-slots is already a flag; host RAM is the budget (64 GB)	Medium: host-Adam step cost already measured +2.3%/step; watch drive RAM ceiling	Strong
5	Distill from a downloaded open teacher (reverse-KL, on-policy on byte rollouts, or plain logits KL at first)	Distillation dominates supervised pretraining for students when a teacher exists; pick a teacher near ~2.5x dormouse size (arXiv:2502.08606, arXiv:2311.07052, arXiv:2306.13649)	Medium: teacher logits on byte streams; tokenizer bridge work	Medium: vocab mismatch, teacher is BPE-based; byteify-style bridging exists (arXiv:2512.15586)	Strong (for the mechanism), medium for the byte-bridge
6	Muon+ everywhere it applies with update-RMS matching and weight decay on matrices; do not hand-tune LR separately	~2x compute efficiency vs AdamW compute-optimal (arXiv:2502.16982)	Already done in optim.rs; verify the RMS-matching constant	Low	Strong
7	Fine-grain the MoE: more, smaller experts at fixed active count, token-choice routing, no shared expert	OLMoE: 64/8 > 8/8, matches dense 7B at <half FLOPs (arXiv:2409.02060)	Low-medium: config + routing code	Medium: more experts = more TSCT factors to retract	Strong for plain experts; untested for TSCT bodies
8	Multiple epochs up to 4 on the filtered corpus instead of one pass	4 epochs ≈ fresh data (+0.5% val loss at 7B); optimum shifts toward epochs when data-constrained (arXiv:2305.16264)	Low: streaming shuffle already seeded	Low; beyond 4 epochs waste	Strong
9	Replace the PonderNet halt head with per-token depth routing (MoR-style) or keep fixed loop depth per iteration; keep the weight-shared loop	MoR Pareto gains at 135M-1.7B equal FLOPs (arXiv:2507.10524); halting-head advantage measured ~0 when readout controlled (arXiv:2608.22347)	Medium: router + partial-forward plumbing; watch the 4D-slicing rule	Medium: changes checkpoint semantics	Strong against halting head, strong for the loop
10	A/B an energy head (InfoNCE over 256 byte candidates) on the loop state before any further aux heads	EBT: ~35% faster scaling, +29% from inference compute (arXiv:2507.02092); doubles as calibration/OOD signal for §3.3 stress metrics	Low-medium: one head + 256 scores/position, same machinery as DSpark	Low at weight 0 default	Medium-strong

What to DELETE (or demote to a one-flag A/B)

• PonderNet halt head as the stopping mechanism. Measured advantage ≈ 0 at controlled readout (arXiv:2608.22347); the field moved to routers or fixed depth (arXiv:2507.10524). Keep the loop, keep a logged halt probe if you like the telemetry, but do not spend training signal on λ prediction.
• KoLeo aux term. Zero public LM-scale evidence; origin is DINOv2 (arXiv:2304.07193). Delete or drop to a permanent-off flag; JEPA alone carries the anti-collapse story (arXiv:2509.14252).
• Output-side lookup/n-gram scaling, forever. Output vocab growth actively hurts small models (arXiv:2501.16975). If anyone proposes Engram-style tables on the logits side, this is the citation to kill it.
• Expert-choice routing. Loses to token-choice at matched compute in OLMoE’s controlled sweeps (arXiv:2409.02060).
• Batch-size warmup. Already flagged by the Qwen3.8-Flash-Next report as wasted under GR+Muon (−18.8% extra steps, no gain); nothing in 2024-2026 contradicts it.
• MSA top-k sparse attention at s512 context, demote pending one A/B. Published sparse-attention wins live at 32K+ contexts (hardware-aligned sparse attention, arXiv:2502.11089); at 512 bytes the quadratic term dormouse is sparsifying is a rounding error, and burn-msa’s gradient_detach leak history shows the arm carries real engineering tax. If the router blend can express “attention ≈ dense here”, measure and keep whichever wins BPB per step-time.
• TSCT low-rank expert bodies, pending a matched-param A/B. Not deletion, but an explicit gate: the entire fine-grained-MoE evidence base (arXiv:2409.02060) uses dense small experts; no public result shows spectral low-rank bodies beating them at equal active params. If the A/B is neutral, prefer plain experts and keep TSCT only where its ortho-maintenance cost is already sunk.
• Sub-4-bit ambitions and fp8-forward-by-default on long runs. The precision laws put the sweet spot at ~4-8 bits with saturating returns by 6-8, and low-precision degradation grows with overtraining, which is precisely dormouse’s known failure mode on small corpora (arXiv:2411.04330). bf16 storage + fp32 accumulation + fp32 heads stays the default; fp8 factors only behind --quant on stable-loss runs.

One process note: rows 1, 2 and 8 cost no architecture work and compound with everything else. The 2024-2026 literature is unusually consistent that at dormouse’s scale, data and measurement discipline out-earn new mechanisms.