Jul 2026· International Conference on Ubiquitous and Future Networks· pp. 765-767· 0 citations· 23 references
Abstract
Large Language Models (LLMs) have brought transformative advancements to natural language processing. However, their massive computational and memory requirements—stemming from tens of billions of parameters—make deployment on resource-constrained edge devices such as mobile phones and embedded systems highly challenging. Consequently, extensive research efforts have focused on enabling efficient on-device LLM inference. This paper provides an in-depth survey of two major inference acceleration techniques for on-device LLMs: KV cache optimization and speculative decoding. We present the core principles, representative studies, and performance characteristics of each approach. Furthermore, we discuss open challenges and future research directions, aiming to contribute to the advancement of efficient on-device LLM systems.
This paper presents a practical evaluation pipeline for deploying small language models (SLMs), compact versions of large language models designed to run with reduced memory and computational requirements, on constrained Android-based edge devices, using Android TV as a representative case. The study investigates both deployment feasibility and software-level acceleration strategies, such as the choice of the inference engine responsible for executing the model on the device, under strict memory and processing limitations. Results show that most models above 500 million parameters were not suitable for the target environment, while a subset of 4-bit quantized models achieved stable execution and acceptable response quality. In addition, the experiments demonstrate that the choice of inference engine has a major impact on performance, with MNN significantly outperforming llama.cpp on ARM-based devices (the processor family that dominates mobile and embedded hardware). These findings provide practical guidance for integrating generative AI into low-resource consumer hardware.
Suayder M. Costa, Igor Lima, William Harada et al.· Anais do XVIII Simpósio Bras...· 0 citations
Evaluation across multiple datasets shows that SelectInfer achieves significant reductions in memory footprint and computation while preserving task performance, making it a practical step towards enabling LLM deployment on edge devices.
Huzaifa Shaaban Kabakibo, Eric Schniedermeyer, Artem Burchanow et al.· 0 citations
This study presents the first comprehensive, cross-layer measurement study of mobile LLM inference, uniquely spanning five mainstream frameworks and three hardware backends, and identifies a distinct phase split where NPUs excel at compute-bound prefilling, while CPUs outperform all other backends in memory-bound decoding.
Guanyu Cai, Ruiming Tian, Lang Yang et al.· 0 citations
Lightweight large language models (LLMs) are increasingly being deployed locally on personal computers and are expected to play a growing role in resource-constrained edge and mobile environments. In such settings, energy consumption, execution time, and memory usage directly affect practical usability, yet existing evaluations of LLM efficiency largely rely on proxy descriptors such as parameter count or FLOPs, often decoupled from task precision. This paper introduces a PTME-based experimental framework for the precision-aware profiling of lightweight LLM inference, jointly measuring Precision, execution Time, peak Memory usage, and Energy consumption through direct hardware-level measurements. The methodology is applied to a representative set of lightweight LLMs executed locally under edge-class resource envelopes on a controlled desktop platform, using benchmarks spanning code generation, mathematical reasoning, and multi-task understanding. We find that static proxy descriptors approximate inference cost well but fail to predict precision. Tightening the resource envelope increases cost without affecting precision, amplifying execution time more strongly than energy and penalizing larger models the most. Moreover, no single model dominates across all PTME dimensions, and a Pareto analysis reveals non-dominated configurations that would be hidden by accuracy-only or efficiency-only assessments, providing practical guidance for selecting models under different resource envelopes. These results show that selecting lightweight LLMs by size, FLOPs, latency, or accuracy alone can select the wrong deployment candidate; PTME profiling exposes configurations that preserve useful accuracy at lower physical cost.
Tomohiro Harada, Enrique Alba, Gabriel Luque· 0 citations
Five self-speculative decoding techniques are characterized across three model sizes and three datasets and recommendations for future research in this area are provided.
Jungmin Ha, Karthik Ganesan, Anh Nguyen et al.· 0 citations
An MLIR (Multi-Level Intermediate Representation) based compilation method for large language models, illustrated using two dialects of operators, TopOp and TpuOp, supporting a variety of generative models including the Qwen, Llama, InternVL, and MiniCPM-V series.
Pengchao Hu, Zhibin Xin, Yifan Chen et al.· 0 citations