Match the Distribution, Not the Compute: Post-Training Multi-Token Prediction Heads
Multi-token prediction (MTP) improves the throughput of autoregressive generation by enabling the language model to draft multiple next tokens per forward pass, while a verification step over draft tokens ensures that token distribution of the backbone is preserved. Every open MTP-family release (MiMo-7B, DeepSeek-V3,...
Prachi Badarayani, Aidan Jay, Cheng-Hui Zhou et al.
· 0 citations