Challenging Quadratic Attention - A Holistic View On the Rise of Alternative Language Model Architectures
This work critically analyze approaches regarding compute and memory complexity, benchmark results, and fundamental limitations to assess whether the dominance of pure-attention transformers may soon be challenged, which it considers possible, particularly in domain-specific and edge-device applications.