Adam Y. J. Jones

Speedbumps: Rejection Attacks on Speculative Decoding

Speculative decoding is a popular technique for increasing the speed and reducing the costs of large language model (LLM) inference by verifying multiple draft tokens in a single target-model forward pass. The resulting benefit depends on the ability of the drafter to approximate the target model's distribution. In thi...