Signed Rescue Routing: Harm-Aware Cascades for Efficient LLM Inference
Signed Rescue Routing is introduced, a budgeted routing method that predicts these two events separately and ranks requests by their difference and shows that predicting incremental value, rather than model uncertainty, is a simple and effective objective for efficient LLM cascades.
Zhe-Yuan Wang, Si-Yu Li, Pei-Qiao Song et al.
· 1 citation