J. Batista

Accelerating Long-Context Decoding in Large-Scale Hybrid LLMs Without Retraining

The full-attention layers of large language models (LLMs) make every generated token more expensive than the previous one, slowing models down on long-context tasks. Hybrid models reduce this cost by replacing most full-attention layers with linear-attention layers, but the first still dominate decoding at long context...