Performance-Efficiency Tradeoffs in Transformers: An Approximation Theory Perspective
This work investigates how the model parameters -- mainly attention heads and head dimensions -- should be allocated across layers to balance expressivity and efficiency, and proposes principled strategies for allocating attention heads and dimensions across Transformers' layers.
Ruoxi Yu, Hao-Tian Jiang, Jingpu Cheng et al.
· 1 citation