dormouse post-training — self-evolving + EGGROLL

Цель: 1B становится ×2 умнее на той же базе (как GLM-5.3), all-bf16.

Loop

A[Pretrained] → B[SFT] → C[Agent rollouts hard envs] → D[Verifier+anti-hack]
D→|pass| E[RLVR PPO+value] → H[On-policy distillation selective] → I[Checkpoint]
D→|fail| F[Failure memory] → G[Improve task/tool/verifier/curriculum] → C
I → J[Holdout] →|beats| K[Promote] →|no| L[Strategy review] → G

SFT (policy init, 30-50% expert, 30-50% synthetic verified, 10-20% repair, 10-25% chat)

RLVR — PPO + value head (not GRPO)

Reward shape for token thrift (MiMo-V2.6, 2610.11959 + 2609.32577)

• Multiplicative: R = R_test · S_sol · S_beh (test pass × solution score × behavior score) — a zero in ANY factor cannot be bought back by the others.
• Group-relative length penalty (2609.32577’s shape, gated): LP = 1 + α·(⌈quantile(passing_lengths, β) − len⌉ / q) — only on PASSING group members, only when the group pass rate ≥ γ; the α/β/γ args are OURS to fit. MiMo’s own live page shows length grew 1.9–2.1× in 30 steps even WITH the penalty — expect a slowdown, not a stop.
• Prompt-mean loss aggregation (not token-mean): a terse correct answer counts as much as a verbose one; byte-thrift enters the loss, not just the reward.
• Grader economics: MiMo buys credit via grader compute (12.7% pass = 1 in 8 needs judge reuse). At our scale the default is the deterministic verifier (code exec, exact match) + the heuristic quality proxy S_beh — a learned/LLM judge is a later, measured node.
• Router FREEZE is their measured zero-cost fix (CV 0.78→2.0, cold experts 0.5%→22%); our LFB integrator covers the same failure mode at ≤1B — freeze only if A/B says so.

Byte-thrift contract (owner directive)

correct@budget(bytes) — accuracy FOR a byte budget — is a first-class metric on every RL/eval report; “solved it, cheaply” is part of the success criterion of the self-evolve loop.

• Value head V(s) bf16, GAE λ0.95, PPO clip 0.2, KL 0.01
• reward = verifier_pass − λ_tokens·T − λ_cost·tools
• Token-level advantage, not group-relative (GRPO is DPO, encourages verbosity)
• EBRM (2504.13134): energy-score уточняет reward на инференсе/в loop — robustness к reward-hack, кандидат после первого PPO-прохода

On-policy distillation — selective (SEED 2607.14777)

• Student rolls, teacher gives dense token guidance only on visited states
• Cross-family → selective, not KL

Self-evolving (core)

Evolves: weights, memory, skills/tools/prompts, verifiers, tasks, harness, curricula.

1. Task-time: K=8 noisy rollouts (PonderNet Q-head) + pick by Q; апгрейд — EBM-реранкер (Residual-EBM, Meta / EBT 2507.02092): segment-level energy по verifier-лейблам, best-of-n без изменений базовой модели
2. Post-task: success→skill, fail→diagnostic → memory
3. Stage-wise: every 2k steps verified trajectories only → PPO+distill → new harness

Taxonomy: Self-Evolving Coding Agents 2608.03392 (framework/memory/skills/tools/workflow/environment)

Store: parquet trajectory + reward/tokens/tools/verifier log → CurateEvo selection (verified success, near-miss, diverse, high info gain)

EGGROLL — exploration for controllers

2511.16652: rank-1 ES, σ0.001, r1, antithetic pairs, ternary sign(s+−s−), α0.001, seed in-place.

• Mutates: PonderNet halt, MoD/MoR routers, MSA budget
• Update: M ← M + (α/N) Σ E·f via diag(f)AᵀB, no materialize
• PPO exploits policy, EGGROLL explores routers

Env pools (GLM-5.3 style)

• code: Tmax + WebGrader
• math: GSM8K/MATH exact
• agent: Qwen-AgentWorld state-prediction + tools All tasks: synthesis → solvability → anti-reward-hack screening

Curriculum

• mastered 5% (regression)
• frontier 70% (pass@k 30-60%)
• too hard archived

CLI

dormouse train --base latest.bin --data mix --preset small
dormouse post-train --base sft.bin --rl --env code,math --verifier hard  # PPO+value
dormouse harness evolve --memory skills --curate