Prasanna Parthasarathi

DimPO: Dimensionality Reduction for Attention using Preference Optimization

A linear projection can reduce the dimension of query and key vectors without updating the pretrained model, but it remains unclear which training objective best preserves model behavior. We ask whether preferences over keys and attention mass on the highest-weighted keys provide a better signal than matching the full...