Multi-Head Self Attention is a Parameter Identification Mechanism
It is proved that a multi-head scaled dot product attention can be viewed as a parameter identification strategy, and a purely mathematical/statistical explanation, identification, for why specific architectural choices in transformers may have improved performance is discussed.
W. R. Morrow
· 0 citations