Yong-Chao Long

When Does Better Acceptance Pay? A Controlled Study of Speculative Decoding

The throughput gains of speculative decoding depend on the trade-off between acceptance length and execution latency. We ask when additional acceptance is worth the cost of obtaining it. We establish a comparison protocol spanning training, inference, and evaluation, remeasuring released checkpoints and retraining six...