What this is

What is this

• Byte-level: vocabulary of 256 bytes. No BPE, no tokenizer tax on Russian, code, or anything else the tokenizer did not see in training.
• Mixture-of-Experts with dropless routing: 1024 experts, top-k 2, no capacity drops — every token keeps every expert it routed to.
• KDA linear attention (Kimi-style chunked gated delta rule, via FLA): O(n) context, a fixed 256 KB recurrent state at any context length — no KV cache to blow up.
• TSCT low-rank experts: every expert is U·diag(s)·Vᵀ (rank 64), which is what lets a 1.07B-parameter model fit, train, and ship on 16 GB.
• Engram memory: 402M hash-table “parameters” that live in host RAM, outside the compute graph.
• A weight-shared loop: the same block re-run per iteration; the presets run four. The depth A/B (small protocol, 2k steps, 81 920 B window) put three ahead by 0.0042 BPB against a control spread of 0.0016, and the mixture runs’ own depth curve is still improving at four (depths=1:1.89,2:1.66,3:1.62,4:1.61 at step 7500 of small_wide) — so depth is an open question the presets answer with the maximum, not a settled optimum.
• The full pipeline in one CLI: train → bench → sft → rl → serve, with a knowledge base rendered from the repo’s own docs.

The presets: small (9.5M, the A/B polygon), medium (138M, the current workhorse), b1 (1.07B total / 57M active, the flagship).