DABO is proposed, a calibration-aware binary offloading method for collaborative large–small model inference that maintains competitive end-to-end accuracy while processing an average of 83.72% of requests at the edge.
Abstract
Large-small model collaboration, as an emerging paradigm for efficient language model inference, plays a significant role in balancing inference quality and service efficiency in cloud-edge intelligence systems. By combining the strong generalization capability of cloud-side large language models (LLMs) with the low-latency inference capability of edge-side small language models (SLMs), this paradigm provides a practical way to serve heterogeneous language requests. However, existing collaborative inference methods still face a critical challenge: forwarding all requests to the cloud leads to high latency and unnecessary service cost, while relying only on edge-side SLMs may degrade prediction quality for difficult inputs. Therefore, we propose DABO, a calibration-aware binary offloading method for collaborative large–small model inference. Specifically, the edge-side SLM first performs lightweight probe inference and produces a prediction confidence distribution. DABO then calibrates the confidence through temperature scaling and converts it into a label-space-normalized difficulty score. Based on a model–task-specific operating threshold selected under an explicit accuracy constraint, reliable requests are answered locally by the edge-side SLM, whereas uncertain requests are selectively escalated to the cloud-side LLM. Extensive experiments demonstrate that DABO maintains competitive end-to-end accuracy while processing an average of 83.72% of requests at the edge, reducing expected calibration error by 31.08%, and decreasing mean latency by 84.24% compared with Cloud-only inference in a controlled workstation-based edge simulation.
A Mobile Reasoning-as-aService (MORES) framework that treats reasoning as a computational service accessible to edge devices over wireless networks, and focuses on implicit reasoning, which achieves an approximately 18% improvement in system throughput over the baseline Soft Actor-Critic (SAC) algorithm.
Guanchen Liu, Hongyang Du, Kaibin Huang· 1 citation
Edge inference has emerged as a promising solution for the proliferation of artificial intelligence (AI) services by deploying models at the network edge to circumvent cloud-routing latency. Existing edge inference approaches mainly focused on either cooperative inference to reduce latency or lightweight model design to fit resource-constrained devices. These solutions often address the communication and computation challenges separately, and thus struggle to achieve a balanced trade-off among transmission efficiency, on-device processing cost, and inference accuracy. To bridge this gap, this paper proposes a hybrid-precision task-oriented communication framework for edge inference to holistically balance communication, on-device computation, and utility. In this framework, a binarized front-end is deployed on the edge device to extract and transmit binary features via orthogonal frequency-division multiplexing (OFDM) signals, while a full-precision back-end on the edge server performs the final inference. To ensure model consistency, we introduce an on-device binarization method tailored for split inference and develop an integrated channel-aware transmission scheme featuring subcarrier-based feature calibration. Furthermore, a knowledge distillation (KD)-based training strategy, supported by specialized gradient estimators, is developed to optimize the end-to-end system and inherit semantic knowledge from a full-precision teacher model. Extensive experiments on the large-scale ImageNet dataset demonstrate the superiority of the proposed hybrid system. Our analysis confirms that this design achieves an optimal trade-off among communication efficiency, on-device computational cost, and inference accuracy, outperforming existing edge inference solutions.
Songjie Xie, Wei Guo, Shenghui Song et al.· 0 citations
Results show that learned token-level handoffs can reduce LLM use while preserving strong reasoning performance, and show that learned token-level handoffs can reduce LLM use while preserving strong reasoning performance.
Niqi Lyu, Pengtao Shi, Wei Qiu et al.· 0 citations
This work presents a distributed inference framework that integrates speculative decoding across edge and cloud, and shifts the bulk of computation to the edge, significantly lowers inference time and cloud cost, and preserves the accuracy of the big model without any retraining requirement.
D. J. Bajpai, K. Upadhyay, M. Hanawal· 0 citations
Results show that DEFRAG narrows the SLM-LLM accuracy gap, while reducing cost by up to 98.4% and increasing peak throughput by up to 97.8% over centralized services, demonstrating the potential of DEFRAG for democratized LLM services at the edge.
Jiaxing Li, Hengzhi Wang, Feng Wang et al.· IEEE Transactions on Mobile...· 0 citations
Modern recommendation models gain prediction quality by scaling feature-interaction and sequence modules, but production cost constraints cap how far systems can scale. In this work, we propose Request-Oriented Compute Sharing (ROCS), a modeling and inference paradigm that exploits a unique property of recommendation inference: each user request is evaluated against many candidates, while request-side features are shared across candidates. ROCS defers request-candidate interactions as late as possible, isolates candidate-dependent representations, and evaluates substantial portions of the model once per request rather than once per candidate, significantly improving inference efficiency while maintaining or improving prediction quality. To realize this paradigm, we develop Generalized Layer Masking (GLM) to enforce candidate isolation in feature-interaction architectures, and Deep Cross Attention (DCA) to extend request-oriented sharing to sequence architectures. To support efficient GPU deployment, we co-design In-Kernel Broadcast Optimization (IKBO) that significantly accelerates ROCS model execution. Experiments on public benchmarks show that ROCS consistently improves the quality-efficiency tradeoff across recommendation backbones. On production-scale workloads, ROCS achieves up to a 3x QPS improvement on retrieval models without quality degradation and a 0.5% relative LogLoss improvement with a 50% QPS gain on a short-form video ranking model. ROCS has been deployed across large-scale recommendation systems spanning ads and organic surfaces, retrieval and ranking stages, and more than two orders of magnitude in inference complexity, delivering significant online gains at reduced infrastructure cost.
Yuxin Chen, Liang Luo, Buyun Zhang et al.· 0 citations