Asymptotic Behavior of Transformers
It is shown that even a three-token, single-layer attention-only transformer can fail to converge, and that, while each transformer has a temperature threshold, the supremum of this set is infinite, so there is no global minimal temperature that forces all transformers to converge.
Katy Blumer, Da-Cheng Juan, Jon M. Kleinberg et al.
· 0 citations