SMat-Attention: Structured Long-Context Sequence Modeling
Long-context sequence models face a fundamental tradeoff: softmax attention uses flexible token-level interactions at quadratic cost, whereas linear attention obtains linear-time training and constant-time decoding by compressing history into a fixed-size state. In this work, we ask whether we can connect these regimes...
E. Anand, Abdullah Ateyeh, Archer Wang et al.
· 1 citation