Dich Nhat Minh Nguyen

MOIRA: Mass-Oriented Indexing with Ragged Attention for Long-Context Decoding

Long-context decoding is limited by memory bandwidth, because every output token reads the KV cache of every layer. Sparse decoding reduces this cost by reading only part of the KV cache. We observe that the number of pages a query needs varies widely across KV heads, layers and steps. Fixed budgets are simple, but the...