The model walk

Architecture

src/dormouse/
├── config.py        strict schema; unknown keys and unported arms refuse loudly
├── instrument.py    the LOUD / COUNTED / SILENT degradation layer
├── model/
│   ├── model.py     DormouseModel: embed → loop → norm → head
│   ├── loopblock.py the weight-shared loop; adaptive depth; MSA gate
│   ├── kda.py       KDA attention (FLA chunked kernel; SDPA fallback, counted)
│   ├── experts.py   dropless MoE: router, LFB balancing, padded/Triton dispatch
│   ├── tsct.py      U·diag(s)·Vᵀ expert factors + the storage ladder (fp8→ternary)
│   ├── engram.py    hash memory, in-VRAM or host-RAM (bit-identical)
│   ├── byteflow.py  the byte patcher (local→global→decoder)
│   └── sparse/      MSA: block-sparse attention + the fused Triton gather
├── train/           loop, WSD schedule, Muon+AdamW, clip, SFT, mixture streaming
├── post/            verifiers (code/math), rollouts, CRR advantage, the RL stepper
├── bench/           downstream eval: ARC-Easy, HellaSwag, synth math (byte-scored)
├── selfplay/        the zero-data self-play core (UTM + gradient-alignment reward)
└── cli.py           train · bench · sft · rl · serve · decode · plot · info

Three things are custom because nothing in the ecosystem does them: the TSCT quantized experts, the Engram host tables, and the ByteFlow patcher. Everything else — attention, optimizer, quantization formats, fused kernels — is the best available dependency, and the places where that choice was made against a homegrown alternative are recorded in docs/.

FAQ

Can this really train on one consumer GPU? Yes. Every run in Results was taken on one RTX 5060 Ti (16 GB, $429): the expert bank is factorized to rank 64, the 402M-row n-gram table lives in host RAM, and the loop is weight-shared, so 1.07B stored parameters cost ~13 GiB at b8 × s512.

What is a byte-level model? The vocabulary is the 256 byte values. Any language, source code or binary file is already in the alphabet, so there is no tokenizer to train and no tokenizer mismatch on the languages it did not see. The price is fewer characters per forward step than a subword model; this project measures that price instead of hiding it.

How is this different from a small Llama or Qwen? The recipe, not the size class: a weight-shared loop instead of depth, dropless top-k Mixture-of-Experts instead of a dense FFN, KDA linear attention with a fixed 256 KB state instead of a KV cache, and hash-table memory outside the compute graph. The published comparison is in the scorecard, with the protocol gap named.

Is it a chatbot? Not yet. The pipeline has SFT and a verifier-first RL stepper (train → bench → sft → rl → serve); the checkpoints are research models, and the site publishes which numbers were retracted and why.

Does this project collect any data about me? No. The site has no analytics, no cookies and no third-party requests: the one script (htmx) is vendored in this repository, the fonts are the system’s, and the demo model is a file served from the site itself. The trainer writes logs to your disk and nothing leaves it.