Stratified Sparse Attention: Reducing the Sparse-to-Dense Gap at Extreme Sparsity
Large Language Model inference over long contexts is increasingly bottlenecked by the cost of attending to a growing Key-Value (KV) cache. Sparse attention reduces this cost by accessing only a fraction of the cache, but existing methods face a trade-off between deterministic retrieval, which captures high-scoring toke...
Sahil Joshi, Prithvi Dixit, Aryan Katiyar et al.
· 0 citations