Grouped Value Attention: Efficient KV Caching via On-Demand Key Reconstruction
This work introduces Grouped Value Attention (GVA), which stores grouped values and reconstructs content keys with a learned linear map at inference, which can be absorbed into the query, eliminating the need to materialize content keys in the intended decode path.
Vishesh Tripathi, Abhay Kumar, Ramsha Khan
· 0 citations