Bharat Runwal

Triadic Linear Attention: Three-Dimensional Recurrent States for Long-Context Sequence Modeling

This work proposes triadic linear attention, which writes the triadic outer product of a key, a second key, and a value, into a third-order tensor state, and reads from it by contracting both key axes with two queries.