CASQ: Configuration-Aware Sensitivity for Mixed-Precision Quantization of Vision Transformers

Abstract

Deploying vision-transformer backbones on edge devices requires low-bit quantization, and mixed-precision methods must assign each layer a bit-width under a memory budget. Allocation is usually driven by per-layer sensitivity estimated from loss curvature (Hessian traces), following HAWQ. We propose CASQ (Configuration-Aware Sensitivity Quantization), which measures sensitivity directly: it samples 150 random joint bit assignments, measures the accuracy each configuration loses with all layers quantized simultaneously, and fits an additive damage model whose per-layer coefficients feed the same integer-linear-program allocator as every baseline. On ImageNet-1k with DeiT-tiny, DeiT-small and ViT-Base/16, the fitted model predicts held-out damage with a coefficient of determination of 0.82 to 0.95, and CASQ is the best or within 0.7 top-1 points of the best allocator at eleven of twelve model-budget pairs, exceeding curvature-based allocation by up to 9 points on ViT-Base/16 while matching it on DeiT-small. The advantage tracks how poorly curvature ranks measured damage on a given architecture rather than model size and narrows to the most aggressive budgets once output-aware calibration is used. On a real-time detector (DEIMv2, COCO mAP), CASQ with 450 configurations gives the best allocation at the aggressive budgets, while uniform quantization is preferable at the loosest budget, for reasons we explain mechanistically.