H-Spec: Parallel Speculative Decoding Without a Drafter-Side KV Cache
Speculative decoding losslessly accelerates large language model inference by having a lightweight draft model predict future tokens for verification by the target model. Recent block diffusion drafters further reduce drafting latency by predicting multiple tokens in parallel. However, existing block drafters project t...
Wei-Fan Jiang, Krishna Teja Chitty-Venkata, Megan Flynn et al.
· 0 citations