From Attention Sensitivity to Layer Role: Revisiting Mixed-Precision Quantization of Transformers
This work defines one scalar loss over the joint Q, K, V weights of a block, evaluated against the block's real causally-masked attention output, and uses it twice: to fit the quantized weights, and to score the block for a multiple-choice knapsack allocation.
Nafiseh HosseinpourFardi, Negar Alihadi, Mahmoudreza Babaei et al.
· 0 citations