Hybrid Gated Attention
The proposed HyGA sheds light on a more effective, efficient, and stable attention mechanism that contains three types of gating strategies and introduces low-rank matrix decomposition and learnable attention sink to further enhance training efficiency and stability.
Zekun Zhou, Ruo-Bing Xie, Lan-Rui Wang et al.
· 1 citation