Skip to content

DimPO: Dimensionality Reduction for Attention using Preference Optimization

Sep 2026 · arXiv.org · Vol abs/2609.32579 · 0 citations · 55 references
Computer Science

TL;DR

The introduction of DimPO, which combines listwise preference optimization with a lightweight top-k cross-entropy term for head-fidelity, suggests that under dimensionality reduction, preserving the ordering and concentration of task-relevant attention can matter more than reproducing the full attention distribution.

Abstract

A linear projection can reduce the dimension of query and key vectors without updating the pretrained model, but it remains unclear which training objective best preserves model behavior. We ask whether preferences over keys and attention mass on the highest-weighted keys provide a better signal than matching the full attention distribution with KL divergence, especially in long-context settings. We introduce DimPO, which combines listwise preference optimization with a lightweight top-k cross-entropy term for head-fidelity. DimPO is trained offline from the attention patterns of a frozen language model, with one map per layer, shared by the query and the keys and trained separately from the other layers. Across LLaMA3.2-3B, LLaMA3.1-8B, Qwen2.5-7B, and Qwen3-4B Instruct models, pairwise preference objectives outperform the triplet baseline and retain 98% of the original score on short-context tasks when projecting to half the dimension on the last 40% of the layers. With more projected layers or on long-context RULER, they degrade rapidly. In contrast, KL and DimPO, which use every key during training, retain about 95% of the original RULER 4k score on the 8B model when projecting up to 50% of the layers. KL-based projections remain closer to the original attention distribution and attention output, yet DimPO achieves better downstream performance. Beyond 50% of projected layers, DimPO increasingly outperforms KL on tasks including SQuAD, common-word extraction, frequent-word extraction, and variable tracking. These results suggest that under dimensionality reduction, preserving the ordering and concentration of task-relevant attention can matter more than reproducing the full attention distribution.

View source

Similar papers

#natural language process... Preprint Oct 2026

Sparse Attention Is Matrix Approximation, Not Choosing from a Bag of Values

It is argued that sparse attention should be formulated as matrix approximation, not as blindly choosing the largest values from a bag of entries, and proposed Matrix Approximation Sparse Attention (MASA), which replaces raw attention-mass ranking with a closed-form score that measures how much each sparse unit reduces...

Fang Wan, Xu-Feng Liu, Fan Li et al. · 0 citations
#machine learning Preprint Sep 2026

Retrieval Capacity of Self-Attention Under Competition

How many tokens from its context does a language model actually use, and what determines that number? We study this question through self-attention. Without retraining, we retain only the tokens with the highest attention weights at each head, layer, and query, keeping their original weights unchanged. By varying the s...

Timur Mudarisov, M. Burtsev, Radu State · 0 citations
#natural language process... Preprint Sep 2026

SAS: Simple Attention Sparsification via End-to-End Optimization of Context Ranking

Across reasoning, long-context understanding, and agentic tasks, SAS consistently outperforms trainable sparse attention baselines across attention budgets, with especially large gains under tight budgets, demonstrating more effective context ranking for downstream tasks.

Zhi-Wei Li, Lei Zhu, Hao Gu et al. · 1 citation
#machine learning Preprint Sep 2026

When Can Attention Heads Be Statically Defined?

Some attention heads learn similar patterns across inputs. Reusing these patterns could reduce training cost by avoiding repeated query-key score computation and softmax. Through controlled pretraining comparisons, we identify Selective Attention Freezing (SAF), which selects heads with low attention-pattern variance a...

Weixian Waylon Li, Yin-Tao Tai, Marcio Fonseca et al. · 0 citations
Preprint Sep 2026

Generative Late-Interaction Embeddings For Visual Document Retrieval

Generative Late-Interaction Embeddings (GLIE): k<<N vectors per page learned from the normalized centroids to serve as both a lightweight index and a basis for regenerating the page's full embedding set, with the decoder as its main design surface.

Mohamed Eltahir, Talal Aloushan, Rose Khairoalsendi et al. · 0 citations
Preprint Aug 2026

Giga-Embeddings: Mixture-of-Experts Encoders for High-Throughput Text Embeddings

This work introduces Giga-Embeddings, a family of text embedding models designed to combine strong retrieval quality with efficient serving, and trains the compact model using a dimension-agnostic objective that aligns teacher and student similarity distributions.

Egor Kolodin, Egor Krasnoperov, Evgeniy Kosarev et al. · 0 citations

Related blog posts

Microsoft Research Blog Nov 16, 2026

TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient

AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.