Skip to content

Beyond Reconstruction Loss in Post-Training Quantization: Balanced Fitting for Large Vision-Language Models

Sep 2026 · arXiv.org · Vol abs/2609.34765 · 0 citations · 45 references
Computer Science

TL;DR

Experiments on multiple LVLMs show that the Balanced Fitting method consistently outperforms prior PTQ approaches under both weight-only and weight-activation quantization, while lower reconstruction loss does not reliably translate into better downstream performance.

Abstract

Post-training quantization (PTQ) enables efficient deployment of large vision-language models (LVLMs), but is typically calibrated on a small set while expected to generalize across diverse downstream tasks. Although recent PTQ methods for LVLMs incorporate sensitivity signals, they still minimize reconstruction loss with respect to the full-precision model, potentially over-preserving FP behavior and calibration-specific bias. Rather than treating quantization solely as an error to be minimized, we observe that it can also provide beneficial regularization for certain layers and modalities. Motivated by this observation, we propose Balanced Fitting, a quantization effect-based framework that balances precision and regularization beyond reconstruction-based optimization. By measuring layer- and component-wise quantization effects for weights, vision activations, and text activations, Balanced Fitting combines fine-grained fitting for sensitive components with coarser fitting to exploit potential regularization benefits. Experiments on multiple LVLMs show that our method consistently outperforms prior PTQ approaches under both weight-only and weight-activation quantization, while lower reconstruction loss does not reliably translate into better downstream performance. The source code is publicly available at https://github.com/kmc3661/BFQ

View source

Similar papers

#machine learning Preprint Oct 2026

Understanding the Weight Averaging Mechanism in LLM Training for Post-Training Quantization

Large language models (LLMs) are typically pretrained in high precision but increasingly deployed with low-precision post-training quantization (PTQ). Recent studies have shown that using weight averaging during pretraining can improve PTQ performance compared with learning-rate decay, suggesting that it might provide...

Han Wang, Tianqi Shen, Zong-Lin Liu et al. · 0 citations
Sep 2026

GLF-Q: Global-Local Feature-based Quantization for Vision Transformers

Extensive experiments across representative ViT architectures demonstrate that GLF-Q with standard uniform quantizers substantially outperforms state-of-the-art methods under 3-bit quantization on image classification and exhibits strong out-of-domain calibration robustness and achieves speedups under 8-bit GPU deploym...

Pei Sun, Guang-Qi Liang, Jin-Nian Tong et al. · 0 citations
#natural language process... Preprint Aug 2026

QUASAR: Lowering the Loss Floor of Quantization-Aware Training with Loss-Aware Reconstruction

QUASAR is introduced, a QAT method that continuously performs lightweight, loss-aware reconstruction in the training loop to lower the loss floor and improve the resulting low-bit model, establishing QUASAR's objective as a principled optimization target.

Vincent Counathe, Ben Athiwaratkun, C. De Sa et al. · 1 citation
Preprint Aug 2026

Activation Outliers Matter: Robust Recovery for Quantized Multimodal LLMs

This work proposes Residual Fallback Quantization (RFQ), a lightweight activation reconstruction framework that supplements the primary ulta-low-bit activation representation with an auxiliary quantized residual pathway that improves activation fidelity while preserving the efficiency advantages of ultra-low-bit comput...

Tanzila Rahman, Mehran Taghian Jazi, Yunke Peng et al. · 0 citations
Conference Open access Sep 2026

Isotropic Embedding Perturbations for Robust Vision Language Encoders

Aether is introduced, a simple plug-in method that applies diffusion-style random perturbations in the embedding space via controlled alpha-mixing, specifically designed to provide isotropic regularization that remains semantically consistent.

Hyesong Choi, Daeun Kim, Song Park et al. · 0 citations
Open access 2026

BiReg: Bilateral Regularized Kernel Adaptation for Training-Free Few-Shot Adaptation of Vision–Language Models

Bilateral Regularized Kernel Adaptation (BiReg), a closed-form, training-free framework that formulates few-shot VLM adaptation as multi-output kernel ridge regression, is proposed that achieves superior accuracy and robustness while retaining the efficiency of a closed-form, training-free solution.

Omar Arif, Aizah Arif · 0 citations

Related blog posts

MIT News · Artificial Intelligence Nov 17, 2026

Learning the language of machine learning

By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.

Microsoft Research Blog Nov 16, 2026

TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient

AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.