FTC, a sequential structured compression framework that adapts the approximation to the current compressed model while jointly exploiting the native Q/K/V head structure under a fixed storage budget, achieves the lowest WikiText-2 perplexity among the compared methods.
Abstract
Post-training compression of LLM attention is often formulated as independent matrix approximation, ignoring both the shared structure among attention projections and the representation shift introduced by earlier compression. We propose FTC, a sequential structured compression framework that adapts the approximation to the current compressed model while jointly exploiting the native Q/K/V head structure under a fixed storage budget. The output projection is handled separately to account for the changed post-attention representation. FTC requires neither fine-tuning nor gradient-based recovery. Across seven decoder-only LLMs from 6B to 32B parameters, FTC achieves the lowest WikiText-2 perplexity among the compared methods at every tested keep ratio on five modern GQA models, with the largest gains under aggressive compression. The improvements transfer to downstream tasks and remain substantial at the 32B scale.
A Context-to-Answer-Aligned Memory Compression (CMC) framework is proposed, which compresses long input contexts into compact Context Memory Embeddings aligned to any frozen decoder's embedding space, reducing inference costs without modifying decoder weights.
Md. Mostafizer Rahman, Md. Faizul Ibne Amin, Md. Shahajada Mia et al.· Conference on Empirical Meth...· 1 citation
TTR-Compressor is evaluated as the compressor for block-wise sparse attention and the visual token merger for multimodal large language models and improves RULER performance over attention-based compressors by a large margin.
Sihan Yang, Jie Huang, Han-Xuan Li et al.· 0 citations
Large language models have shown strong performance across many natural language tasks, but their size makes deployment difficult on hardware with limited memory and compute. Compression techniques aim to reduce this cost while preserving as much quality as possible. This study investigates structured pruning applied t...
Adnane Bakkou, Badr Hirchoua, M. Banane et al.· International journal of adv...· 0 citations
Linear attention enables efficient long-context autoregressive decoding by compressing history into recurrent states, but this compression can make selective access to sparse and distant information difficult. Existing chunk-based extensions increase memory capacity, yet learned chunk-mixing coefficients may remain fix...
Zhuo-Kun Chen, Xi Lin, Xi-Yu Wu et al.· 0 citations
Standard attention forms query (Q), key (K), and value (V) matrices through learned linear projections of input hidden states. Architectures such as GAU and Mega incorporate nonlinear transformations into attention. In this paper, we propose GatedV, a simple attention variant that applies SiLU only to V before aggregat...
The full-attention layers of large language models (LLMs) make every generated token more expensive than the previous one, slowing models down on long-context tasks. Hybrid models reduce this cost by replacing most full-attention layers with linear-attention layers, but the first still dominate decoding at long context...