dormouse

A byte-level Mixture-of-Experts language model, trained end to end on one consumer GPU.

dormouse(1)rtx 5060 ti · 16 gb
$ dormouse train medium --set seq_len=1024 --set moe_capacity=0
step=60000  loss=1.2130  bpb=1.936  40.9k tok/s
$ dormouse serve --ckpt-name sft11 --port 3783
ready — one RTX 5060 Ti 16GB, $429
[+] 968 tests[+] ruff clean[-] cuda gates self-skip
1.07Btotal params · 57M active
40.9ktok/s, medium b24 × s1024
1.936held-out BPB, 196,608 B window
968tests, ruff clean
$429one RTX 5060 Ti 16 GB

Why this exists

The frontier publishes its recipes and not its compute. This project asks the narrower question: take the same ideas — linear attention, sparse experts, memory outside the weights, low-rank factors, bytes instead of tokens — and point them at one $429 GPU. How far do they actually go?

The framework is the instrument; the model is the reading. Every number on this page was taken on this box, at a named config and commit, and a mechanism that fails its A/B is recorded as a failure rather than deleted quietly.

Admitted and rejected by measurement

Status: the measured numbers, retractions included

The five axes

byte-level

Vocabulary of 256 bytes. No BPE and no tokenizer tax on Russian, code, or anything the tokenizer never saw in training.

KDA linear attention

Kimi-style chunked gated delta rule. O(n) context and a fixed 256 KB recurrent state at any context length — there is no KV cache to blow up.

dropless MoE

1024 experts, top-k 2, no capacity drops: every token keeps every expert it routed to.

TSCT low-rank experts

Every expert is U·diag(s)·Vᵀ at rank 64. That factorization is what lets a 1.07B-parameter model fit, train and ship on 16 GB.

Engram memory

402M hash-table parameters that live in host RAM, outside the compute graph.

The stack, and why each piece is here

byte-level input
one vocabulary for every language, code and binary; no tokenizer to train, and speculative decoding accepts 3.4× more tokens than subword (2610.05978).
KDA attention
linear in length with a fixed recurrent state, so a long context costs memory that does not grow — the reason 1M context is a configuration, not a rebuild.
dropless MoE + LFB
capacity that does not drop tokens: load balancing by a bias on the router, not by an auxiliary loss that fights the language objective (2408.15664).
TSCT low-rank experts
a 1.07B-parameter model in 16 GB: each expert is U·diag(s)·Vᵀ at rank 64, and the same factorization carries the fp8→NVFP4→ternary storage ladder.
Engram memory
knowledge that costs no FLOPs: 402M hash rows in host RAM, of which one token touches ~0.02%.
weight-shared loop
depth without parameters — three passes through one block, the measured optimum of 2/3/4 on our protocol.
Muon + ColRow, WSD
the best of five optimizers A/B’d here; NorMuon, schedule-free Muon and SIGReg all lost to it and are recorded as losses.
verifier-first post-training
code and math verifiers plus fork-replay advantages — post-training without a learned reward model, and a length penalty that rewards solving in fewer bytes.
QAT ladder
training happens in bf16 and the release is a rung of the same model: 0.3 GB ternary against 2.1 GB bf16, measured by the same eval.

A knowledge base with a retraction policy

Every page of the docs is generated at build time from the file that already lives in the repository — no second copy of any fact. A number is published with the config, the date and the commit it was taken at, and a claim that was taken back stays published next to the one that replaced it.

What this is ·Retracted, do not cite ·The glossary

Peek the docs without leaving

Three pages of the knowledge base, loaded in place by htmx from the same static build — no second copy of any fact, no reload. Without JavaScript the same links navigate normally.

[+] status[+] what this is[+] glossary

Pick a page — it loads here from the same build. Full navigation still works: each is a real link.

Quick start

uv sync                      # CPU environment; the tests run here
uv run pytest                # 968 tests, ruff clean
uv run dormouse train medium --data corpus.bin --eval held.bin --device cuda \
  --steps 60000 --batch 24 --set seq_len=1024 --set moe_capacity=0
uv run dormouse serve --out-dir checkpoints/sft --ckpt-name sft11 --port 3783

Requires uv and Python 3.12 — the full list of commands.

questions people actually ask

Can this really train on one consumer GPU?
Yes - the runs in the table above were taken on one RTX 5060 Ti (16 GB). The model that fits is a byte-level Mixture-of-Experts: 1.07B parameters stored, ~57M active per byte, with the 402M-row n-gram table in host RAM and the expert bank factorized to rank 64.
What is a byte-level model, and why no tokenizer?
The vocabulary is the 256 byte values, so any language, code or binary is already in the alphabet and there is no tokenizer to train or to mismatch. The cost is that a forward pass reads fewer characters per step than a subword model would; the project measures that cost and reports it rather than hiding it.
How is this different from a small Llama or Qwen?
The recipe is different, not the size class: a weight-shared loop instead of depth, dropless top-k Mixture-of-Experts instead of a dense FFN, KDA linear attention with a fixed 256 KB state instead of a KV cache, and hash-table memory outside the compute graph. The published comparison is in the docs, with the protocol gap named.
Is it a chatbot?
Not yet. The pipeline has SFT and a verifier-first RL stepper, and the demo shows the architecture running in the browser; the trained models are research checkpoints, and the site says which numbers were retracted and why.
Does the site track me?
No. No analytics, no cookies, no third-party requests: the one script (htmx) is vendored in this repository, the fonts are the system ones, and the demo model is a file served from this site.