DimPO: Dimensionality Reduction for Attention using Preference Optimization
The introduction of DimPO, which combines listwise preference optimization with a lightweight top-k cross-entropy term for head-fidelity, suggests that under dimensionality reduction, preserving the ordering and concentration of task-relevant attention can matter more than reproducing the full attention distribution.