Deploying large language models (LLMs) for inference on edge devices is challenging due to severe memory and bandwidth constraints. While speculative decoding and Mixture-of-Experts (MoE) have been proposed to improve inference efficiency, naively combining them often incurs excessive verification overhead and poor expert reuse, limiting their effectiveness in memory-bound edge settings. In this work, we propose S2-MoE, an efficient self-speculative decoding framework for MoE inference on edge devices. S2-MoE reduces redundant verification through routing-aware adaptive speculative expansion, improves verification efficiency with reuse-aware expert gating, and aligns draft and target execution via shared context. Implemented in llama$.$cpp, S2-MoE achieves up to $5.3\times$ speedup (about $2.0\times$ on average) over standard autoregressive decoding across diverse MoE models and datasets on edge devices. Code is available at https://github.com/angerybob/S2-MoE.
DraftExpert is proposed, an expansion-aware self-speculative decoding framework for expert-offloaded MoE inference that improves decode throughput by 1.45x on average, raises draft acceptance to 84~87%, and achieves 86~88% prefetch hit rates.
EdgeXpert is proposed, a software-hardware co-designed LLM accelerator that resolves this incompatibility and achieves up to 56.3% latency reduction and 44.1% energy reduction compared to prior works, while maintaining near-baseline accuracy.
Sangwoo Ha, Hyunwoo Seo, Yurim Jo et al.· 0 citations
This work formulate expert-cache management as a model-side algorithmic problem and proposes cache-aware post-training that jointly adapts the MoE backbone and lightweight auxiliary routers while preserving the native inference-time Top-K rule.
Zhen-He Wu, Ya-Ping Jin, Qing-Hua Xing et al.· 3 citations
Five self-speculative decoding techniques are characterized across three model sizes and three datasets and recommendations for future research in this area are provided.
Jungmin Ha, Karthik Ganesan, Anh Nguyen et al.· 0 citations
A unified efficiency analysis is presented showing that extending the speculation horizon can reduce rather than improve speedup when the marginal acceptance probability falls below the relative drafting cost, and SparseSpec-L, a training-free self-speculative decoding framework for long-context inference is introduced...
Yue Liu, Yuan Zeng, Min Lyu et al.· arXiv.org· 0 citations
EPIC mitigates imbalance via performance-aware expert migration and runtime expert activation, and then improves communication with topology-adaptive transport kernels and fine-grained computation-communication overlap.
Jia-Min Cao, Qingxu Li, Yaozhong Liu et al.· Conference on Applications,...· 2 citations· ⚡1
Machine-learning networks can learn without detailed instructions from a designer. Those that do may offer insight into how the brain reorganizes itself during learning.
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
