The rapid scaling of large language models (LLMs) has unlocked remarkable capabilities across various domains. However, deploying LLMs on mobile devices remains challenging because of their high computational and memory requirements. Low-bit quantization (e.g., 4-bit weights, 8-bit activations) can reduce model size wh...
Hang-Yu Zheng, Zhi-Hao Shu, Kun-Xiong Zhu et al.· Proceedings of the 32nd ACM...· 1 citation
EAServe repositions Encode as the control point of the EPD pipeline, exposing three tightly coupled dimensions: when work enters downstream, where prefill executes, and how the GPU is shared, and reaches near-optimal configurations faster than baseline search methods.
Kun-Xiong Zhu, Zhi-Hao Shu, Hang-Yu Zheng et al.· 0 citations
TierKV is presented, a mobile LLM inference framework built on Predictive Multi-Tier Cache Optimization (PMCO), which improves prefill throughput by up to 17.6x over existing mobile LLM frameworks, reduces RAM-resident KV cache by 12.5-34%, thereby enabling substantially longer contexts under the same memory budget, wh...
Zhi-Hao Shu, Md Musfiqur Rahman Sanim, Jie Hu et al.· arXiv.org· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.