DWI: Efficient On-Device LLM Decoding Through Dynamic Width Selection on Nested Models

Abstract

Deploying large language models (LLMs) on resource-constrained mobile devices remains a significant challenge due to memory limitations and high I/O overhead. While existing dynamic sparsity methods reduce computation costs, they often suffer from inefficient loading patterns that degrade I/O throughput. To address this, we propose to leverage dynamic width selection and optimized parameter management on nested models to enable efficient on-device LLM inference. Our approach introduces three key techniques: 1) an end-to-end dynamic width selection mechanism that optimizes nested LLMs’ width configurations across tokens to balance efficiency and quality, 2) a cyclic-locality-aware cache policy that dynamically evicts stale parameters to minimize redundant I/O, and 3) a block-wise load-computation pipeline that overlaps in-memory computation with loading from flash to mitigate I/O latency. Experimental results demonstrate that our method accelerates on-device LLM inference by up to <inline-formula><tex-math notation="LaTeX">$34\times$</tex-math><alternatives><mml:math><mml:mrow><mml:mn>34</mml:mn><mml:mo>×</mml:mo></mml:mrow></mml:math><inline-graphic xlink:href="cao-ieq1-3711495.gif"/></alternatives></inline-formula>, while maintaining competitive model performance, making it a promising solution for on-device LLM deployment.