PEQuant: Prompt-Driven Activation Reconstruction for Elastic Quantization of Vision Transformers

Abstract

Elastic precision quantization enables efficient multi-bit deployment through a single optimization process. However, extending it to Vision Transformers (ViTs) remains challenging due to high optimization costs and unstable activation distributions with extreme outliers, which severely disrupt multi-bit optimization. More importantly, existing elastic post-training quantization methods for ViTs remain largely weight-centric, leaving activation reconstruction underexplored. To address this issue, we propose PEQuant, a framework for error reconstruction at Transformer block inputs. By applying a Stacked Low-Rank Additive Prompt, PEQuant performs activation-level reconstruction without modifying quantized weights while introducing only negligible sequence overhead. To stabilize this reconstruction under low-bit quantization, we introduce a lightweight Artifact-Absorbing Extension Prompt that suppresses severe outliers and high-norm artifacts that would otherwise destabilize prompt learning. Furthermore, to support robust joint multi-bit calibration, PEQuant adopts a cascaded low-rank parameterization with Multi-Precision Activation Fusion. This design allows higher-bit branches to guide lower-bit prompt learning while preventing aggressive low-bit updates from contaminating higher-bit representations. Consequently, PEQuant naturally complements existing weight-space methods, requires no weight optimization, and transfers effectively to downstream tasks without additional fine-tuning. Extensive experiments demonstrate that PEQuant consistently improves the accuracy of various quantized ViTs.