Do Dynamic Routers Need Memory? HeRo: History-Aware Routing for Efficient LLM Inference
History-Aware Routing is proposed, a dynamic routing framework that resolves this mismatch by introducing a router memory mechanism to maintain an explicit routing state across model depth, validating that explicit routing memory enables more accurate and adaptive dynamic routing than solely conditioning on hidden stat...