Skip to content

Osprey: Target-agnostic Pre-training Makes Stronger Drafters in Speculative Decoding

Sep 2026 · 0 citations · 63 references
Computer Science

Abstract

Speculative decoding is critical for accelerating LLM inference. However, the speedup is fragile: drafters are typically trained against a narrow distribution for a single target model, and their acceptance rate collapses under workload shifts. This is a striking inversion of modern LLM development, where target models are valued precisely for the broad generalization they acquire through large-scale pretraining. We argue that the natural remedy, pretraining, has been hard to apply to drafters because existing recipes are target-specific: the drafter consumes the target's hidden states and is distilled on the target's logits, so pretraining must be repeated for each target. We introduce Osprey, which instead bootstraps drafters from off-the-shelf pretrained small language models, treating broad pretraining as a reusable, target-agnostic asset and reducing per-target work to a lightweight adaptation step. Realizing this requires overcoming two challenges: small LMs are far deeper than a latency-bound drafter can afford, and their pretrained computation must remain intact while the drafter learns to ingest target hidden states and emit tokens in the target's vocabulary. Osprey addresses both by pruning to a shallow backbone, restoring its language-modeling capability with target-agnostic next-token pretraining, and adapting it to each target through vocabulary alignment, zero-initialized QKV expansion, and distillation from the target model's output distribution. Empirically, a single pretrained Osprey backbone transfers across targets and improves mean acceptance length by 16.1% for Qwen3-8B, 21.2% for Llama-3.3-70B-Instruct, and 22.7% for the 229B MiniMax-M2.5 (with 17.5% higher tokens per second), with the largest gains on out-of-domain and multilingual data. Our code is available at https://github.com/LeanModels/Osprey.

View source

Similar papers

Preprint Sep 2026

HAWK: Rethinking Multimodal Drafting for Speculative Decoding

Speculative decoding has achieved substantial lossless speedups for LLMs, but remains less effective for large vision-language models (LVLMs), where lightweight drafters struggle to use rich multimodal information. A second limitation is that standard distillation supervises the drafter only along the original training...

Wen-Han Yang, Anirudh Rao, Ashwin Chandra · 0 citations
#artificial intelligence Preprint Oct 2026

T2D: Target-to-Draft Model Pruning&Adaptation for Efficient Speculative Decoding

Speculative decoding significantly accelerates autoregressive text generation in Large Language Models (LLMs), but its efficacy depends largely on two key factors: the distributional alignment between the draft and the target model, and the draft model's latency during inference. However, existing approaches typically...

Shahriar Kabir Nahin, Xiao Ye, Mu-Hao Chen et al. · 0 citations
#natural language process... Preprint Oct 2026

TreeSpark: Learning to Draft the Trees You Verify for Speculative Decoding

TreeSpark is introduced, a unified training and inference framework extending the semi-autoregressive DSpark architecture that constructs parent-conditioned candidate trees from one backbone pass and replays their expansion, pruning, and budgeted packing during training.

Jihao Zhao, Qian-Yu Jiang, Shuai-Shuai Zu et al. · 1 citation
Preprint Nov 2026

Give the Drafter a Second Chance to Guess: Parallel Reconsideration for Speculative Decoding

Speculative decoding accelerates large language model inference with a small drafter, whose per-guess acceptance rate determines the achievable speedup. Feature-level drafters share a design choice: every token, easy or hard, is drafted under a single forward pass. Many of the tokens the target rejects are ones the dra...

Hoang Anh Duy Le, Hui-Bing Dong, Alireza Khadem et al. · 0 citations
Sep 2026

LongSpark: Efficient speculative decoding with a fixed-cost parallel drafter

Speculative decoding accelerates autoregressive inference by verifying multiple draft tokens in a single target forward pass. However, as the context grows, existing state-of-the-art drafters become increasingly expensive, eroding the very efficiency advantage they are designed to provide. We argue that this scaling is...

Hao-Yuan He, Peng-Fei Liu, Chen Chen et al. · 1 citation
Preprint Oct 2026

RCSD: Reinforcement and Continual Learning for Domain-Specific Speculative Decoding

Speculative decoding accelerates large language model inference by employing a compact draft model to propose candidate tokens that are verified in parallel by a target model. Conventional training relies on soft-label distillation, which forces the draft model to mimic the target's full output distribution, including...

Ming-Kang Liu, Wei-Ming Meng, P. Jiao et al. · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.