Skip to content

Calibrate the Decisions That Change the Future: On-Policy Post-Training Quantization for Multimodal Large Language Models

Sep 2026 · 0 citations · 33 references
Computer Science

TL;DR

OnPTQ is proposed, an on-policy framework that calibrates on trajectories visited by the current quantized policy, and derives a Decision--Consequence bound linking behavioral deviation to current policy discrepancy and action-conditioned future-value span.

Abstract

Post-training quantization (PTQ) lowers deployment cost for multimodal large language models, but calibration typically reconstructs fixed sequences with local objectives. This overlooks autoregressive feedback: a quantization-induced token change redirects the prefix and changes future states. Yet on-policy coverage alone is insufficient because many decision mismatches barely affect future generation. We propose OnPTQ, an on-policy framework that calibrates on trajectories visited by the current quantized policy. On shared prefixes, OnPTQ identifies quantization-eroded boundaries, evaluates competing tokens through short counterfactual rollouts, and combines current discrepancy with branch consequence into a Decision--Consequence risk. The risk prioritizes critical states, while context anchoring and trajectory refresh preserve multimodal behavior and keep calibration aligned with the updated policy. We further derive a Decision--Consequence bound linking behavioral deviation to current policy discrepancy and action-conditioned future-value span. Across vision--language and omni-modal Qwen models under multiple low-bit settings, OnPTQ improves downstream performance and yields fewer correctness flips against the corresponding Dense/FP16 references, without changing the deployed inference graph.

View source

Similar papers

#natural language process... Preprint Aug 2026

Recovering General Capabilities via Uncertainty-Calibrated Multi-Teacher On-Policy Distillation

The proposed Uncertainty-Calibrated MOPD (UCMOPD), which addresses limitations of standard MOPD through two complementary mechanisms, improves the general-capability average over standard MOPD by $4.48\% and $7.86\% while maintaining vertical-domain performance.

Zi-Yuan Liu, Jiao Ou, Jian Liang et al. · 0 citations

Predict Before You Deploy: Offline Prediction of Quantization-Induced Task Degradation for World Action Models

PreDE (Predict Before You Deploy), a policy-calibrated framework for predicting quantization-induced task degradation from offline action deviations, supports policy-specific behavioral calibration for quantization configuration selection while identifying candidates that require closed-loop evaluation.

Jiu-Yi Xu, Jin-Jia Guo, Mei-Da Chen et al. · 0 citations
#machine learning Preprint Sep 2026

From Weak Data to Strong Policy: Q-Targets Enable Provable In-Context Reinforcement Learning

Q-Target Pretrained Transformers is proposed, which keeps the context-conditioned Transformer architecture but replaces behavior cloning with a Bellman-style Q-target objective, and learns to use rewards and transitions in the context to estimate action values, rather than simply imitating the behavior policy.

Yi-Chen Lin, Xu-Yuan Xiong, Xue Wang et al. · 0 citations
#machine learning Preprint Oct 2026

Cross-Behavior Policy Optimization: Unified Learning from Offline Demonstrations and On- and Off-Policy Rollouts via a Shared Regression Target

Reasoning post-training draws on offline demonstrations and policy-generated rollouts, yet conventional supervised fine-tuning (SFT) followed by reinforcement learning (RL) separates their objectives and limits continued reuse of collected responses. We propose \textbf{Cross-Behavior Policy Optimization} (CBPO), which...

Du Cheng, Jun-Yu Lu, Jing-Fan Zhang et al. · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.