Speculative decoding accelerates memory-bound LLM decoding with a lightweight drafter, and parallel drafters extend this speedup by drafting a whole block in one forward pass. However, it remains unclear whether their released systems (stock) attain the full speedup that parallel drafting permits. We identify three cha...
Kun-Jun Li, Wen-Hao Chai, Yihao Liang et al.· 0 citations
LeapQuant is proposed, a training-free method that achieves near-lossless performance under 8-bit recurrent-state quantization and substantially reduces memory and compute costs during inference.
This framework organizes existing methods around five components: explicit state representation, transition structuring, validation and constraint enforcement, search and rollback, and uncertainty management, and yields a diagnostic hypothesis: interventions should be most effective when they target the error or uncert...
Jia-Zhang Cai, Tao Wang, Rui-Dong Zhang et al.· arXiv.org· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.