Strong Drafts Need Compact Memories: Long-Context Speculative Decoding with Compressed KV Cache
Memory-augmented drafting is introduced for long-context SD, equipping a strong independent draft with compressed draft-side KV memory and incrementally updates this memory to retain distant information and exact recent context.
Tong Yuan, Cheng-Xi Liao, Ze-Yi Wen
· 0 citations