#artificial intelligence
May 2026
More Expressive Feedforward Layers: Part I. Token-Adaptive Mixing of Activations
This work proposes Mixture of Activations (MoA), a token-adaptive FFN design that mixes a dictionary of activation functions using lightweight input-dependent gates while sharing the same linear projections, suggesting that token-adaptive activation mixing is a simple and effective mechanism for improving FFN expressivity in LLMs.
Mingze Wang, Jinbo Wang, Yikuan Xia et al.
· arXiv.org · 3 citations