Attention-Aware Routing: Coupling Routing and Attention in MoEs
In Mixture-of-Experts language models, the router typically selects and weights experts based on the token's hidden state, utilizing limited contextual information. We propose Attention-Aware Routing (AAR), which augments the router with temporal and spectral features extracted from a sliding window of attention weight...
Despoina Kosmopoulou, Anastasios Tsetsilas, Efthymios Georgiou et al.
· 0 citations