Empowering edge intelligence through reparameterized lightweight transformers and distributed inference
TL;DR
A reparameterized transformer framework that integrates High-Rank Factorization (HRF) during training, layer merging at inference, and dynamic, load-balanced distributed inference across multiple devices is proposed, highlighting that reparameterized transformers, coupled with adaptive distributed inference and ultra-low-precision execution, offer a compelling solution for real-time, on-device analytics in IoT and edge computing environments.
Abstract
Deploying advanced transformer-based models on resource-constrained edge devices remains a significant challenge due to their high memory footprint and substantial compute requirements. In this paper, we propose a reparameterized transformer framework that integrates High-Rank Factorization (HRF) during training, layer merging at inference, and dynamic, load-balanced distributed inference across multiple devices. To further reduce resource usage, our framework supports mixed-precision quantization down to 4-bit, enabling flexible accuracy–latency–energy trade-offs. Experimental evaluations on the ESC-50 environmental sound dataset demonstrate that our method matches or exceeds the performance of larger baseline models while using 20–30% fewer parameters, achieving up to 48% latency reduction in multidevice setups, and substantially lowering energy consumption. Ablation studies confirm the benefits of tuning factorization rank, partition strategy, and quantization level for diverse edge scenarios. Overall, our findings highlight that reparameterized transformers, coupled with adaptive distributed inference and ultra-low-precision execution, offer a compelling solution for real-time, on-device analytics in IoT and edge computing environments.