Speculative decoding accelerates large language model inference through parallel verification of tokens proposed by a drafter. Diffusion drafters further parallelize proposal generation, but greater proposal capacity or higher mean accepted tokens (MAT) need not yield higher throughput. Because verification accepts onl...
Jia-Kun Fan, Ming-Qing Xiao, Qiang Wang et al.· 0 citations
This work proposes preserving shared prefixes and evicting the KV cache from the beginning of the remaining history while retaining the cached suffix and builds Penelope around this strategy by co-designing the eviction policy, KV recomputation, and unified GPU-CPU cache management.
Jia-Kun Fan, Han Meng, Yu-Xin Xia et al.· 0 citations
Diffusion Large Language Models (dLLMs) have emerged as a promising alternative to Autoregressive Models (ARMs), utilizing parallel decoding to overcome sequential bottlenecks. However, existing research focuses primarily on kernel-level optimizations, lacking a holistic serving framework that addresses the unique memo...
Jiakun Fan, Yang-Lin Zhang, Xiangchen Li et al.· International Conference on...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.