Complex KDA: Understanding and Enhancing the Expressivity of Kimi Delta Attention
This work shows that Kimi Delta Attention (KDA) can realize 2D rotations by combining a single delta-rule transformation with a second reflection supplied by its channel-wise gate, and characterize the expressivity of CKDA and prove that every orthogonal diagonal-plus-rank-one matrix is exactly a CKDA transition matrix...
Julien Siems, Riccardo Grazzi, Korbinian Pöppel et al.
· 1 citation
· ⚡1