Dec 2026· Proceedings of the ACM CoNEXT 2026 Student Workshop· 0 citations
TL;DR
This work presents CoordSpec, a framework that coordinates cloud verification scheduling with adaptive draft length selection, and presents the system design and an evaluation plan under varying task loads and network conditions.
Abstract
Speculative decoding (SD) accelerates large language model inference by generating candidate tokens with a lightweight draft model and verifying them with a larger target model. This separation naturally enables edge-cloud inference, where workloads continuously move between edge drafting and cloud verification. In this setting, efficient operation depends not only on the efficiency of individual speculation rounds but also on maintaining a balanced workload flow between the edge and cloud. Specifically, an imbalance in this flow can either underutilize cloud verification or delay verification feedback until an edge exhausts its ready work and becomes idle. We present CoordSpec, a framework that coordinates cloud verification scheduling with adaptive draft length selection. The cloud scheduler prioritizes requests whose delay is likely to cause edge idle while admitting additional requests to improve verification efficiency, and the draft controller adjusts draft lengths to maintain sufficient edge-side work accounting for the accepted progress of each draft. We present the system design and an evaluation plan under varying task loads and network conditions.
Speculative decoding (SD) speeds up large language model (LLM) serving, but its gain depends on the workload's \emph{shape}, such as batch size and context length, and on its \emph{content}. A model-based drafter proposes a short block on any content, while a model-free drafter such as n-gram lookup proposes long copie...
CoSEI is presented, a system that introduces operator-aware partitioning and synchronization-minimizing execution, guided by a lightweight planner and a tailored runtime, that demonstrates that with the right design, collaborative edge inference can make LLMs practical beyond the cloud.
Fei Zeng, Feng Lyu, Hao Wu et al.· IEEE Transactions on Mobile...· 0 citations
Speculative decoding reduces sequential Target model calls by verifying multiple tokens from the Draft model in parallel. Yet KV Cache growth limits long-context serving under constrained GPU memory. Offloading KV to CPU memory relieves this pressure. However, existing offloading schemes restore the full KV history bef...
Fei Li, Song Liu, Shi-Qiang Nie et al.· arXiv.org· 0 citations
Serverless inference is emerging as an efficient deployment model for Large Language Models (LLMs) due to its ability to elastically scale resources, improve GPU utilization through resource sharing, and reduce infrastructure costs. However, existing approaches primarily optimize time to first token (TTFT) while overlo...
Hardi Ashishbhai Desai, K. BalajiN, Sourav Kanti Addya· 0 citations
NebastianSD is presented, a many-for-many, or M-for-N, speculative decoding system that organizes draft and target workers into independently schedulable resource pools and dynamically reconstructs stage-specific batches from shared request pools.
The deployment of Large Language Models (LLMs) on resource-constrained edge server clusters introduces a critical systems challenge: achieving high throughput and low latency under strict GPU memory and bandwidth limits. Current schedulers are ill-suited for this environment, as they treat requests independently, incur...
Chen Chen, Lei Jiao, R. Mortier· IEEE Transactions on Paralle...· 1 citation