A comprehensive benchmark of 20 teachers from 11 well-known ViT families reveals that Attention Transfer is not universally effective, and identifies architectural mismatch between the pre-trained teacher and the standard student as the primary mechanism.
Huai-Yuan Qin, Mu-Li Yang, Gabriel James Goenawan et al.· arXiv.org· 3 citations
Copying MLPs can serve as an effective foundation for Softmax-to-linear transfer: paired with the distilled attention, linear ViTs eventually close the remaining gap and even surpass Softmax ones.
Huai-Yuan Qin, Mu-Li Yang, Gabriel James Goenawan et al.· 1 citation
This work proposes CADE (Contrastive Adaptive Debias Ensemble), a training-free, plug-and-play method that leverages modality-specific answer priors that yields significant gains on the proposed benchmark, which can foster the development of more fair and reliable AI systems for sustainable development.
Zihang Lin, Huaiyuan Qin, Mu Yang et al.· arXiv.org· 0 citations
DiD is introduced, a label-free conversion method that exclusively trains the linear-attention backbone by aligning detector-facing interface tensors with those of a frozen Softmax teacher, and substantially outperforms established baselines and matches supervised, fully trained linear models.
Huai-Yuan Qin, Gabriel James Goenawan, Zihang Lin et al.· 1 citation
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
