Hybrid Speculative Decoding in RAG: When Do the Pros Outweigh the Cons?

Abstract

Speculative decoding emerged as a way to break the sequential autoregressive decoding of Large Language Models (LLMs), based on the idea that a lightweight model produces draft completions, to be efficiently verified in parallel by the actual LLM. Building on recent work where a hybrid speculative decoding strategy was proposed for retrieval-augmented generation (RAG) applications, we thoroughly analyze what makes such hybrid work. Results show that higher token budgets work in favor of hybrid methods. Additionally, we find that the performance varies considerably over different benchmarks, and propose an approximate diagnostic to estimate the performance of such hybrid setup upfront. Finally, we show that the use of a prefix-tree implementation for variable-length n-gram matching is $1.4\times $ faster than reported alternatives. For reproducibility, we release our code, to our knowledge the first public tree-merging implementation for hybrid speculative decoding (github.com/Gianni-Van-de-Velde/hybrid-speculative-decoding).