Gianni van de Velde

Hybrid Speculative Decoding in RAG: When Do the Pros Outweigh the Cons?

Speculative decoding emerged as a way to break the sequential autoregressive decoding of Large Language Models (LLMs), based on the idea that a lightweight model produces draft completions, to be efficiently verified in parallel by the actual LLM. Building on recent work where a hybrid speculative decoding strategy was...