Sparse Attention for Dense Open-Vocabulary Prediction in CLIP
This work studies an inference-time substitution of the row-wise softmax in the final visual self-attention layers with the $\alpha$-entmax transform, applied across both the standard query-key attention and self-correlation variants.