Difficulty-Aware Hybrid Routing for Cost-Efficient Long-Context LLM Inference
TL;DR
The Difficulty-Aware Hybrid Router is proposed, a lightweight framework that jointly leverages query-level, context-level, and draft-level features to classify each request into one of three inference tiers: direct answering by a small model, small-model answering with lightweight verification, or full processing by a large model.
Abstract
Large language models (LLMs) achieve strong performance on long-context tasks but incur substantial inference costs that scale with input length. Existing routing approaches reduce costs by dispatching queries to smaller or larger models, yet they estimate difficulty from the question text alone and overlook context-level factors such as evidence dispersion and similarity entropy. We propose the Difficulty-Aware Hybrid Router (DAHR), a lightweight framework that jointly leverages query-level, context-level, and draft-level features to classify each request into one of three inference tiers: direct answering by a small model, small-model answering with lightweight verification, or full processing by a large model. Experiments on LongBench and RULER demonstrate that DAHR reduces inference cost by 38% at the main operating point and up to 46% under stricter budgets, while maintaining competitive quality and outperforming baselines across context lengths from 4K to 128K tokens.