On attention heads and bilinear forms
It is proved that the symmetric part of a bilinear form in an attention head is the sum of a hyperbolic form and a zero form for a Zariski-dense subset of query-key matrices.
Andrew O'Desky
· 0 citations