This work presents AsymSpec, which addresses uplink-gated verification and invalid dependent work with two corresponding mechanisms, and shows that AsymSpec delivers 2.82-28.03$\times the output-token throughput of the strongest baseline.
Abstract
Cloud-edge speculative decoding places a lightweight draft model at an edge gateway and a higher-quality target model in the cloud, but inserts communication into every speculative block. Under a constrained uplink, candidate messages may queue while the verifier is idle. Stop-and-wait scheduling leaves edge compute underutilized; optimistic same-request runahead can waste work when a rejection or an unexpected bonus token invalidates dependent drafts. We present AsymSpec, which addresses uplink-gated verification and invalid dependent work with two corresponding mechanisms. Its asymmetric verification protocol keeps the common-path acceptance upload compact and moves richer, rejection-only correction information to the downlink. A total-variation (TV) certificate for the residual distribution determines whether a small target top-K response suffices; if not, the protocol progressively escalates through proposal-based exact recovery before falling back to the full distribution. Its confirmed-prefix pipeline exposes only independent, valid requests to the edge scheduler and lets the cloud re-batch arrived blocks, hiding verification waits when another confirmed-prefix request is ready without using same-request runahead. Across three draft-target pairs, two workloads, and three asymmetric network profiles, our end-to-end evaluation shows that AsymSpec delivers 2.82-28.03$\times$ the output-token throughput of the strongest baseline.
Deploying large language models (LLMs) on edge nodes enables low-latency and privacy-preserving inference, but faces severe resource constraints under high-concurrence workloads. While existing inference systems leverage intranode key–value (KV) caching to improve efficiency, they largely neglect the unique complexities of multinode edge environments. Specifically, reactive KV cache eviction policies suffer from temporal uncertainty, often discarding reusable KV caches prematurely, while the tight coupling between request scheduling and cache placement often leads to myopic decisions that exacerbate load imbalance and resource contention. To address these challenges, we propose a dynamic block-level paradigm that treats KV blocks as the fundamental units for caching and scheduling, enabling dynamic sharing, generation, and eviction of arbitrary-length prefixes. We present complete modeling of the spatiotemporal coupling between scheduling and caching under block-level granularity, capturing intricate interactions overlooked by prior work. Based on this model, we design an online joint optimization algorithm, which applies to general edge LLM serving scenarios. The algorithm decouples spatiotemporal dependencies via randomized rounding over per-slot subproblems, achieving a balance between real-time responsiveness and long-term system efficiency. Theoretical analysis establishes high-probability near-optimality guarantees, and extensive experiments show that our method reduces the average time to first token (TTFT) by up to 54.02% over existing baselines.
Xishuo Li, Wei Jiao, Jun He et al.· IEEE Internet of Things Jour...· 0 citations
This work presents a distributed inference framework that integrates speculative decoding across edge and cloud, and shifts the bulk of computation to the edge, significantly lowers inference time and cloud cost, and preserves the accuracy of the big model without any retraining requirement.
D. J. Bajpai, K. Upadhyay, M. Hanawal· 0 citations
Roomie is presented, a model serving orchestration architecture that predicts and avoids kernel-level interference between colocated DNNs, and reduces SLO violations by up to 3x, while maintaining comparable, and in many cases superior, goodput relative to existing approaches.
Youssouph Faye, Francescomaria Faticanti, Shubham Jain et al.· 0 citations
We study reliable query recovery under independent premise erasures in semantically transparent caching systems, where every cached object must be a logical consequence of the premise base. Recovery succeeds only when the query remains derivable from surviving premises and the cache. Under a deterministic canonical-witness regime, we prove a query-local projection theorem and an exact residual-leaf law: recovery fails exactly when an erased base leaf retains a cache-free path to the query. Single-query design becomes weighted partial path interception. For shared workloads, we introduce semantic modules and derive exact reliability laws under joint and maximal-error criteria. The shared-module cache is exactly optimal under exact module routing and homogeneous costs, whereas optimal selection in general derivation DAGs is NP-complete at depth two. Against a coded benchmark recovering workload-relevant leaf payloads, MDS parity caching is optimal up to one packet. Leaf-only transparency incurs a first-order overhead inversely proportional to the erasure rate; shared modules multiply that inverse-erasure-rate scaling by the module-to-leaf cost ratio divided by the number of protected leaves. A Datalog instance and Monte Carlo checks illustrate the theory. For derivation-structured content, the results provide exact stochastic-erasure counterparts of function-correcting storage and an exact distributional quantification of maximal recoverability.
This work presents the first constant-competitive algorithm for arbitrary prompt lengths and arbitrary response lengths with no additional assumptions, built on a novel regime-aware routing framework.
Yiding Feng, Siyu Liu, Zonghan Yang et al.· 1 citation
When affinity recovers too little KV work, its residual load skew reduces or erases the improvement, so gating any deployment with a shadow replay rather than enabling affinity from workload statistics alone is recommended.