This work proposes Gossamer, a distributed scheduler where each edge node routes requests using type-weighted scoring over gossiped partial state, and GLARE (Gossip-driven Local Adaptive Routing Engine), a metaheuristic that uses gossip staleness to control per-node weight adaptation.
An online request scheduling framework for edge LLM inference that jointly minimizes long-term average end-to-end latency and regulates workload distribution across heterogeneous edge servers is investigated and the LYREO approach, a cross-slot inference model that captures transmission, prefill, iteration-level decodi...
Large language models (LLM) are increasingly integrated into pervasive applications spanning heterogeneous cloud-edge resources. These heterogeneous resources differ in computational capacity, energy efficiency, network conditions, and deployed model quality, creating trade-offs among response quality, monetary cost, e...
Xi-Jia Wang, Tian-Zhang He, Mohammad Goudarzi· 0 citations
Large language model (LLM) serving platforms increasingly route requests across heterogeneous backends, transforming query routing from a model-selection task into a system-level load-balancing problem. Centralized routers can become throughput bottlenecks and single points of failure as deployment scale and request vo...
Yuan-Hao Xi, Xiao-Huan Bing, Ramin Yahyapour· Journal of King Saud Univers...· 0 citations
As Large Language Models (LLMs) become essential in privacy-sensitive sectors like hospitals and government agencies, the on-premise LLM servers offer a cost-effective and secure alternative to public cloud services. However, these resource-constrained servers struggle to guarantee heterogeneous Service Level Objective...
Ze-Shen Zhang, Han Zhao, Wei-Hao Cui et al.· 0 citations
Large language model (LLM) routing distributes inference work across different models, advancing the cost-quality Pareto frontier of LLM serving. While coarse-grained routing at the session or query level has been widely adopted in production systems, recent algorithmic work shows that fine-grained token-level routing...
Tianyu Fu, Teng-Xuan Liu, Ruo-Xin Wang et al.· 0 citations
In this paper, we present a latency-aware scheduler for large-language-model (LLM) inference across mobile devices, edge servers, and a remote cloud. Our fine-grained delay model captures OFDMA uplink/downlink rates, KV-cache backhaul serialization, and profiled GPU planning-chunk resource constraints, enabling per-req...
Xinghan Wang, Xiao-Xiong Zhong, Wei-Hong Yang et al.· IEEE Transactions on Paralle...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
