Noise-Contrastive GRPO is introduced, which injects scale-calibrated Gaussian noise into the last hidden layer of the prompt-encoding pass for half of each rollout group, branching those rollouts from a displaced departure state and integrates into a standard RLVR pipeline as a ~50-line change to the inference engine.
Abstract
Reinforcement learning with verifiable rewards (RLVR) improves the reasoning ability of vision-language models (VLMs), and diversifying the rollouts within each optimization group amplifies its gains. Existing approaches diversify through decoding temperature or pixel-space image distortion; we ask whether the perturbation belongs in the model's latent space instead. We introduce Noise-Contrastive GRPO (NC-GRPO), which injects scale-calibrated Gaussian noise into the last hidden layer of the prompt-encoding pass for half of each rollout group, branching those rollouts from a displaced departure state. Branches that reach the answer despite the displacement are reinforced over those derailed by it, converting sensitivity at the branch point into policy-gradient signal; the objective, reward, and inference protocol are untouched. On Qwen2.5-VL-7B trained on Geometry3K, NC-GRPO significantly improves out-of-domain mathematical reasoning over vanilla GRPO across five held-out benchmarks (pooled McNemar $p \le 0.001$) while also improving in-domain accuracy and hallucination robustness -- the latter an axis on which image-space noise regresses even while posting a larger OOD average on perception-heavy benchmarks. Mechanism ablations indicate that independent stochastic diversity, not noise budget or direction, is the active ingredient, and a noise-scale study exposes a dial between reasoning specialization and general capability. NC-GRPO is designed to be modality-agnostic and integrates into a standard RLVR pipeline as a ~50-line change to the inference engine.
Reinforcement learning for vision-language math reasoning starves under sparse reward: on a pool of 20,830 visual-math problems where Qwen2-VL-2B answers 3.6% of rollouts correctly, 85-97% of GRPO rollout groups are entirely wrong and contribute zero gradient. We train eleven methods under identical conditions in this regime, each injecting a different prior: text (reference-solution hints), distribution (on-policy distillation from a 7B teacher), and value (a value-pretrained critic with an MSE or HL-Gauss categorical loss). A prior helps exactly when it is delivered: the six arms whose prior effectively reaches the policy separate with no overlap from the remaining five -- the no-prior baseline and four arms whose prior is teacher-capped, gated away, or lost to a mis-parameterized critic -- both on the pooled in-domain metric and on cross-domain transfer (DynaMath). The central finding, however, concerns evaluation: one slice of the in-domain pool -- long used as this project's general-distribution check -- anti-correlates with genuine cross-domain transfer (Spearman rho = -0.74, n = 11 arms, permutation p = 0.011), while the hardest in-domain slice predicts it closely (rho = +0.89, p<0.001). We attribute the inversion to a near-chance multiple-choice subset that rewards models for not having changed; read through it, the best cross-domain method looked mediocre and the worst looked like the champion. Among the methods, hint-guided exploration -- not UFT's auxiliary loss -- drives hint gains, and replacing the critic's MSE loss with HL-Gauss cross-entropy is worth +14.4 points in-domain. All accuracies are blind-judged, with paired exact tests.
This paper explores post-training reinforcement learning (RL), specifically GRPO, to directly align autoregressive perception models with their evaluation metrics, and designs an RL framework that addresses perception-specific challenges: reward design for set-structured outputs and multi-head sampling control.
This work proposes CritiqueDriveVLM, a novel unified three-stage framework internalizing reasoning directly into the VLM, and proposes Latent Thought Distillation to overcome the latency bottleneck.
Zhaohong Liu, Hao Ye, Xianlin Zhang et al.· 1 citation
Deep neural networks often suffer significant accuracy degradation when exposed to real-world image corruptions and distribution shifts. To overcome the limitations of fixed, input-agnostic test-time augmentation (TTA), an adaptive framework is proposed that learns per-sample transformations via reinforcement learning. Augmentation selection is cast as a Markov decision process and proximal policy optimization (PPO) agents are trained to choose sample-specific transforms under a composite reward combining classifier confidence gains with a self-consistency KL-divergence penalty on the model's own softmax outputs, thereby preserving overall belief stability. On clean CIFAR-10 (1 000 samples), the adaptive ensemble raises accuracy from 88.5% (baseline) and 87.3% (static TTA) to 90.0% (+1.5 pp). On CIFAR-10-C (15 corruptions × 5 severities; 1 000 images per condition), pooled top-1 accuracy improves from 75.7% (baseline) and 74.3% (static TTA) to 76.4% (+0.7 pp), and exceeds a TENT entropy-minimization baseline (75.9%) while operating in a strictly label-free regime that updates no model weights. Per-corruption gains are consistently positive across noise, blur, weather, and compression distortions, with the adaptive policy outperforming TENT on texture and compression corruptions where input-space transforms are most effective. These findings demonstrate that learned, per-sample augmentation policies improve robustness and reliability of deep vision models under diverse image conditions, against a strong baseline classifier.
Tushar Mittal, A. Dubey, Dharmender Saini et al.· Scientific Reports· 0 citations
The sample efficiency and scalability of RL post-training for video MLLMs and introduces OraRL, a decoupled advantage estimator that scales with model size and data, surpassing its backbone from 0.8B to 9B and GRPO up to 100k prompts.
Yunheng Li, Guohong Mu, Hao Li et al.· 0 citations
This thesis proposes a unified two-axis framework that organizes SFT and RL methods along a data axis (off-policy to on-policy) and a loss function axis (positive-only to positive-plus-negative to GRPO) and enables controlled ablations of individual components.
G. Kim, Chair Chenyan Xiong, Aditi Raghunathan· 0 citations