Prism Attention: Proposal-Refined Index Sharing Mechanism for Efficient LLMs Inference
Long-context decoding in large language models (LLMs) is increasingly bottlenecked by the memory and computation required to attend over an ever-growing Key-Value (KV) cache at each autoregressive step. Although token-level sparse attention reduces this burden by restricting computation to a small subset of relevant pa...
Zhen-Xu Tian, Ke-Bin Liu, Zheng-Wu Yang et al.
· 0 citations