The proposed Wireless GPU Computing Infrastructure (WiCi) can reduce time to first token by up to 90%, improve the token rate by approximately 39x compared to local inference on mobile devices for the same model, and support much larger models.
Abstract
LLM inference applications are gaining significant traction. The demand for inference is growing exponentially, and the GPU usage of inference is increasingly surpassing that of training. Due to the mobility penalty, edge-side inference fails to deliver satisfactory performance. Consequently, most inference service providers currently rely on cloud-based inference, which incurs substantial, not sustainable costs for enterprises, and is even increasing in the agentic paradigm. Therefore, our goal is to enable powerful computing capabilities as server-grade GPUs on mobile devices. We propose Wireless GPU Computing Infrastructure (WiCi) in this paper. Through WiCi, mobile devices can wirelessly access server-grade GPUs, running inference tasks on mobile clients but offloading GPU-related computations to a nearby GPU via WiFi. WiCi introduces a series of designs to make sure the infrastructure is scalable with different applications, compatible with different mobile devices, and has comparable performance to running on a physical GPU. We test WiCi from mobile devices and find that WiCi can reduce time to first token by up to 90%, improve the token rate by approximately 39x compared to local inference on mobile devices for the same model, and support much larger models. WiCi also achieves up to nearly 80% of the native performance of the server-grade GPU across different applications.
Computation methods of artificial intelligence are gradually shifting from cloud computing to edge computing and on-device machine learning (ODML). How to contribute an effective machine learning model in the resource-limited environment, has become a significant and rapidly evolving research field. The training and inference of deep learning model used to be performed on the cloud high-performance computing clusters. There are many problems with uploading data to cloud, for example, high latency, round-trip latency, security issues, and a lack of privacy guarantees, and in this case, people cannot make real-time decisions. So, using edge devices to process tasks can significantly decrease the cost of transmission. The need of low latency, quick response, privacy protection and high adaptability has become the drive force of this change. This report aims to provide a comprehensive overview of lightweight model inference technologies in edge computing environments, mainly targeting low performance devices, such as mobile phones, intelligent equipment in vehicle, VR/AR headsets and Internet of Things (IoT). This paper introduces efficient learning and inference on edge devices from four aspects: 1) the definition of core terminology and concrete application environment; 2) the core technology of model compression, neural networks and knowledge distillation, which is used to deal with the tasks in the resource-limited environment; 3) the standards for evaluation of time/space complexity; 4) the challenge and opportunity which people face currently and in future.
Hengrui Bi· Applied and Computational En...· 0 citations
Running large language models on consumer devices such as laptops and desktops is challenging because model weights often exceed GPU memory capacity, making offloading inference necessary to extend effective model capacity with CPU memory. Existing offloading systems, however, typically rely on coarse layer-level or expert-level scheduling, which overlooks substantial heterogeneity among tensors within the same layer and adapts poorly to changing hardware load conditions on such devices. This paper presents ATSInfer, a hybrid CPU-GPU inference system for consumer devices that performs offloading at tensor granularity. ATSInfer combines static tensor placement with load-aware dynamic transfer, and introduces asynchronous CPU-GPU coordination to efficiently schedule hardware storage, data movement, and computation across heterogeneous backends. We implement ATSInfer and evaluate it on representative consumer platforms using both dense and MoE models. Compared with existing systems, ATSInfer improves prefill throughput by up to 1.94$\times$ and decode throughput by up to 3.29$\times$, while also increasing GPU utilization and making more effective use of PCIe bandwidth. These results show that ATSInfer can substantially improve the user experience of local LLM deployment on personal consumer devices.
Yangyijian Liu, Hongyi Ye, Mingyang Li et al.· 0 citations
This paper examines the way phones and IoT devices run deep neural network (DNN) inference with help from a nearby edge server. It uses semantic communication (SC) to shrink the data sent over the air. The work notes that local computing power is limited. It connects two aspects: the level of data compression and the amount of local computation needed to achieve that compression. This link is modeled as a piecewise linear function across DNN layers. Early layers compute less and compress less. Deeper layers compute more and compress more. The objective is to minimize total user energy. The threshold depends on the radio channel quality and the local energy cost per compute cycle. A user above the threshold should upload earlier with less compression. A user below the threshold should compute more locally to compress the data. Next, a low-complexity iterative algorithm is designed for the general case. It iteratively updates the split layer and resources. The proposed scheme saves much more energy than full offloading and then conventional partial offloading. The gains are largest when wireless bandwidth is scarce. The approach scales to many users and suits real mobile edge scenarios. The model is transparent. Assumptions suit edge networks well. It connects theory, algorithms and experiments into one unified framework for energy minimization tasks.
Sumalatha Pilli, U. S. B. K. M. Lakshmi, Thalakola Syamsundararao et al.· ITEGAM- Journal of Engineeri...· 0 citations
A time-varying integer program to minimize the long-term total cost of the edge AI inference system, including the inference latency, the inference error rate, the query-dispatching communication cost, and the energy consumption, subject to resource and workload constraints is proposed.
Mingtao Ji, Hehan Zhao, Lei Jiao et al.· Science China Information Sc...· 0 citations
A reinforcement learning-based computation offloading strategy using an improved Deep Deterministic Policy Gradient (IDDPG) algorithm that outperforms greedy offloading strategies and demonstrates improved efficiency in dynamic MEC environments.
S. Alam, Sami ud Din, Imtiaz Ali Shah et al.· International Journal of Inn...· 0 citations
A scalable edge-to-cloud AI inference pipeline in which inference tasks are dynamically distributed across heterogeneous edge and cloud resources is examined, providing a basis for resilient real-time AI systems while highlighting unresolved challenges involving heterogeneous hardware, dynamic workloads, privacy-utility trade-offs, and cross-layer optimization.
Dr. Khalid Al- Mansour· International Journal of Com...· 0 citations
Related blog posts
MIT News · Artificial Intelligence· news.mit.eduSep 2, 2026
What if pathology foundation models could do more with less? GigaPath-Flash and GigaTIME-Flash cut computational demands while maintaining strong performance, opening the door to larger studies and broader exploration. The post GigaPath-Flash and GigaTIME-Flash: Toward population-scale discovery with efficient pathology foundation models appeared first on Microsoft Research.
MIT News · Artificial Intelligence· news.mit.eduAug 31, 2026
With millions of users across the world, Julia has been used to conduct cutting-edge research and to design new drugs, jet engines, heat pumps, and more.
MIT News · Artificial Intelligence· news.mit.eduAug 27, 2026
A new machine-learning framework aims to improve the success rate of computational protein design while moving away from results that reproduce sequences found in nature.
