Beyond the Previous Layer: Residual Predictive Structure in Sparse MoE Routing
Using frozen OLMoE and JetMoE models, residual predictive structure in expert-selection trajectories beyond adjacent-layer persistence is identified, identifying residual predictive structure in expert-selection trajectories beyond adjacent-layer persistence.