Energy-Efficient Context-Aware Multimodal AI Inference at the Edge

Abstract

Deploying multimodal artificial intelligence models on resource-constrained edge devices faces inherent bottlenecks in energy consumption and computational latency, as conventional full-modality inference pipelines keep all perception encoders active regardless of task context and environmental conditions, causing substantial power waste and degraded real-time performance. To address this challenge, this work presents an energy-efficient context-aware multimodal edge inference framework featuring a lightweight modality activation sparsity evaluation unit, dynamic computation path scheduling, and a cross-modal speculative skipping mechanism. The framework quantifies the information value of each input modality in real time according to scene context, task complexity, and device power status, and adaptively activates or deactivates corresponding visual, audio, and sensor encoders, while tuning model quantization precision and operator fusion strategies to align with runtime resource budgets. Validated on NVIDIA Jetson Nano and Raspberry Pi 5 edge platforms across VQAv2, MMBench, and multimodal perception benchmarks, the proposed framework delivers a 42.3% reduction in end-to-end energy consumption and a 30%-65% decrease in inference latency against static full-modality baselines, alongside $\mathbf{1. 5} \times$ to $\mathbf{2. 3} \times$ higher throughput with an accuracy loss no more than 1.2%. The runtime context scheduling module introduces less than 9 ms of latency overhead and only 0.32 W of additional power draw, with per-inference energy as low as 0.6 J; for battery-powered mobile edge devices, the framework extends continuous operating duration by over 72% under typical perception workloads. These findings confirm that context-aware adaptive scheduling can dramatically boost the energy efficiency of edge multimodal inference without sacrificing task performance, offering a viable deployment solution for low-power Internet of Things, intelligent surveillance, and human-robot interaction scenarios.