Aug 2026· IEEE Transactions on Mobile Computing· 0 citations· 57 references
Computer Science
TL;DR
Results show that DEFRAG narrows the SLM-LLM accuracy gap, while reducing cost by up to 98.4% and increasing peak throughput by up to 97.8% over centralized services, demonstrating the potential of DEFRAG for democratized LLM services at the edge.
Abstract
The rapid advancement of large language models (LLMs) has increased demand for scalable and cost-effective deployment, especially for mobile and edge devices. Cloud-hosted LLMs are powerful but expensive and difficult to scale due to vendor lock-in and high resource needs, resulting in high expenses and unstable performance under load. Recent efforts focus on deploying small language models (SLMs), distilled or pruned from LLMs, on resource-constrained edge devices to reduce costs and improve scalability. However, edge-based SLMs face limited knowledge coverage and notable accuracy gap compared to cloud-based LLMs. To address this, we present DEFRAG, a decentralized edge collaboration system for retrieval-augmented generation (RAG) that optimizes both retrieval and generation across heterogeneous edge devices. For retrieval, DEFRAG compresses and shares knowledge graphs, using hybrid retrieval to expand knowledge coverage. For generation, DEFRAG introduces an optimizer that adaptively selects SLMs and RAG parameters per query, balancing accuracy and cost. We implement DEFRAG on a heterogeneous edge testbed and evaluate it on benchmark QA datasets. We also test it under mobile route stress, non-uniform data placement, and a domain-specific QA workload. The results show that DEFRAG maintains stable service quality and cost efficiency under these broader settings. Results show that DEFRAG narrows the SLM-LLM accuracy gap, while reducing cost by up to 98.4% and increasing peak throughput by up to 97.8% over centralized services. These findings demonstrate the potential of DEFRAG for democratized LLM services at the edge.
This work presents a distributed inference framework that integrates speculative decoding across edge and cloud, and shifts the bulk of computation to the edge, significantly lowers inference time and cloud cost, and preserves the accuracy of the big model without any retraining requirement.
D. J. Bajpai, K. Upadhyay, M. Hanawal· 0 citations
CELLServe formalizes SLO-constrained joint resource provisioning as an optimization problem with a dedicated algorithm, and introduces an opportunistic instance merging strategy for decode phase functions to reclaim fragmented resources.
Zejian Wang, Nan Lin, Zinuo Cai et al.· ACM Transactions on Architec...· 0 citations
DABO is proposed, a calibration-aware binary offloading method for collaborative large–small model inference that maintains competitive end-to-end accuracy while processing an average of 83.72% of requests at the edge.
Chen Zhu, Yi-Ming Su, Chenwenjie Mao et al.· IEEE Access· 0 citations
This paper employs five lightweight machine learning models to predict query-specific latency, accuracy, resource usage, and response size for each model-size-quantization-device combination, and design a lightweight heuristic that approximates the BILP solution.
Reza Farahani, Zoha Azimi, Mario Colosi et al.· 0 citations
Foundation models have improved the reasoning and generation ability of artificial intelligence systems. However, they are difficult to deploy in edge environments with limited computation, memory, and data access. Small models are easier to run on edge devices. They support fast and low-latency inference, but they often lack global semantic reasoning and cross-domain generalization. This gap between model ability and deployment cost motivates large–small model collaboration in cloud–edge systems. This survey provides a systematic review and a knowledge-floworiented taxonomy of such collaboration. It focuses on how cloud-side large models and edge-side small models share, update, and coordinate knowledge. We review knowledge distillation, split inference, federated and continual adaptation, and elastic offloading. We also cover lightweight deployment, modular expert design, privacyaware coordination, and agent-driven orchestration. Unlike surveys on edge intelligence, federated learning, model compression, TinyML, or cloud–edge resource scheduling, this survey centers on model collaboration. We treat large–small collaboration as a knowledge-centered problem linked to real deployment constraints. We further discuss trade-offs in accuracy, latency, bandwidth, privacy, energy efficiency, adaptability, and lifecycle management. Finally, we identify open challenges for trustworthy, sustainable, and self-evolving cloud–edge collaborative intelligence.
Yaxi Yang, Jingye Bi, Haitao Yuan et al.· Journal of Artificial Intell...· 0 citations
Deploying large language models (LLMs) as personal assistants on mobile devices demands privacy, low latency, and offline availability, yet the computational cost of giant models clashes with strict edge-hardware budgets. We argue that this tension cannot be resolved by model compression alone; it requires decomposing on-device intelligence into complementary functional roles. We present SmartRAG, a fully on-device framework that organizes an intelligent assistant around four coordinated modules -- Perception, Memory, Focus, and Thinking. At the core of SmartRAG is EvoNER, a continually learnable named-entity recognizer that incrementally expands its label inventory through teacher-distilled updates, enabling the system to absorb previously unseen entity types without retraining the backbone LLM. Extracted knowledge is stored in MRGraph, a three-layer provenance-preserving knowledge graph, and retrieved at query time through a hybrid pipeline combining graph traversal, lexical matching, and dense semantic search. The on-device LLM is invoked only for high-value semantic operations -- labeling, planning, and answer synthesis -- keeping inference costs bounded. Experiments on four QA benchmarks (TriviaQA, Natural Questions, HotpotQA, MultiHopQA) show that SmartRAG with a quantized 1.7B-parameter backbone achieves multi-hop reasoning performance competitive with models up to 18$\times$ larger, while running entirely on commodity smartphones within practical memory and latency envelopes.
Zhihan Jiang, Meng Li, Shenghao Liu et al.· 0 citations