N. Mitchell

TableTune: Measured, Self-Correcting Draft-Length Tables for Dynamic Speculative Decoding

Speculative decoding speeds up large language model (LLM) serving only while the GPU has spare compute, so the best draft length, the number of tokens a small drafter guesses per step, shrinks as the batch grows. vLLM lets operators set the draft length for each batch size with a hand-written table, but nothing measure...