Skip to content
#large language models Book Open access

CoordSpec: Coordinating Edge Drafting and Cloud Verification for Efficient Speculative Decoding

Dec 2026 · Proceedings of the ACM CoNEXT 2026 Student Workshop · 0 citations

TL;DR

This work presents CoordSpec, a framework that coordinates cloud verification scheduling with adaptive draft length selection, and presents the system design and an evaluation plan under varying task loads and network conditions.

Abstract

Speculative decoding (SD) accelerates large language model inference by generating candidate tokens with a lightweight draft model and verifying them with a larger target model. This separation naturally enables edge-cloud inference, where workloads continuously move between edge drafting and cloud verification. In this setting, efficient operation depends not only on the efficiency of individual speculation rounds but also on maintaining a balanced workload flow between the edge and cloud. Specifically, an imbalance in this flow can either underutilize cloud verification or delay verification feedback until an edge exhausts its ready work and becomes idle. We present CoordSpec, a framework that coordinates cloud verification scheduling with adaptive draft length selection. The cloud scheduler prioritizes requests whose delay is likely to cause edge idle while admitting additional requests to improve verification efficiency, and the draft controller adjusts draft lengths to maintain sufficient edge-side work accounting for the accepted progress of each draft. We present the system design and an evaluation plan under varying task loads and network conditions.

Read PDF

Similar papers

Preprint Nov 2026

Hybrid Spark: Adaptive Speculative Decoding That Lets the Content Choose the Drafter

Speculative decoding (SD) speeds up large language model (LLM) serving, but its gain depends on the workload's \emph{shape}, such as batch size and context length, and on its \emph{content}. A model-based drafter proposes a short block on any content, while a model-free drafter such as n-gram lookup proposes long copie...

Ekagra Ranjan, Bharat Venkitesh · 0 citations
#edge computing Open access Dec 2026

Scaling LLMs via Collaboration: Efficient LLM Inference Across Resource-Constrained Devices

CoSEI is presented, a system that introduces operator-aware partitioning and synchronization-minimizing execution, guided by a lightweight planner and a tailored runtime, that demonstrates that with the right design, collaborative edge inference can make LLMs practical beyond the cloud.

Fei Zeng, Feng Lyu, Hao Wu et al. · 0 citations
Sep 2026

Resource-Efficient Speculative Decoding for Long-Context LLM Serving

Speculative decoding reduces sequential Target model calls by verifying multiple tokens from the Draft model in parallel. Yet KV Cache growth limits long-context serving under constrained GPU memory. Offloading KV to CPU memory relieves this pressure. However, existing offloading schemes restore the full KV history bef...

Fei Li, Song Liu, Shi-Qiang Nie et al. · 0 citations
#artificial intelligence Preprint Oct 2026

Stable Token-level Scheduling for Serverless LLM Inference

Serverless inference is emerging as an efficient deployment model for Large Language Models (LLMs) due to its ability to elastically scale resources, improve GPU utilization through resource sharing, and reduce infrastructure costs. However, existing approaches primarily optimize time to first token (TTFT) while overlo...

Hardi Ashishbhai Desai, K. BalajiN, Sourav Kanti Addya · 0 citations
Sep 2026

NebulaSD: Many-for-Many Speculative Decoding

NebastianSD is presented, a many-for-many, or M-for-N, speculative decoding system that organizes draft and target workers into independently schedulable resource pools and dynamically reconstructs stage-specific batches from shared request pools.

Jun-Hao He, Hong-Yang Du · 0 citations
#large language models Open access Dec 2026

ALPS: An Application-Aware, Length-Predictive Scheduler for Efficient LLM Inference at Edge

The deployment of Large Language Models (LLMs) on resource-constrained edge server clusters introduces a critical systems challenge: achieving high throughput and low latency under strict GPU memory and bandwidth limits. Current schedulers are ill-suited for this environment, as they treat requests independently, incur...

Chen Chen, Lei Jiao, R. Mortier · 1 citation

Related blog posts

Google DeepMind Blog Nov 18, 2026

Genie 4 is now available

We are rolling out a new Project Genie experience to eligible Google AI Ultra subscribers, opening access to our new Genie 4 world model.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.