Skip to content
Log in
Preprint

Enhancing VLM Reward Models Through Structure-Aware Fine-Tuning

Pyrros Koussios Chenhao Li Xin Chen Andreas Krause
Aug 2026 · 0 citations · 41 references
Computer Science

Abstract

Designing effective reward functions remains a major bottleneck in Reinforcement Learning (RL). Recent work uses large foundation Vision-Language Models (VLMs) as reward models, computing text-observation similarity to bypass manual reward engineering. Although promising, these rewards are often noisy and unreliable, limiting their direct utility during deployment. We present Structure-Aware Fine-Tuning (SAFT), a simple, self-supervised method that refines these imperfect reward signals online without access to ground-truth supervision. SAFT leverages intrinsic structural priors to regularize the VLM's latent space via LoRA adapters. We rigorously evaluate SAFT across a spectrum of base model capabilities to demonstrate its versatility. Our results show that SAFT consistently denoises the reward landscape, yielding faster policy convergence and substantially improved alignment (EPIC distance) relative to the underlying base model, suggesting that failures can often be attributed to structural brittleness rather than semantic misunderstanding. By replacing extensive human preference annotation with structural inductive biases inherent to the task, SAFT offers a scalable path for stabilizing text-conditioned RL and underscores the broader value of incorporating task structure as a general inductive bias.

arXiv Semantic Scholar
View source

Similar papers

Preprint Jul 2026

GFlowRL: Scaling Distribution-Matching RL to Large Language Models

GFlowRL, a streamlined GFlowNet-style RL algorithm that removes the auxiliary partition network entirely while preserving the reward-distribution-matching objective, is proposed, and is the first GFlowNet-style RL algorithm to scale stably across both dense and sparse architectures.

Xiaodong Liu, Michael Xu, Jack W. Stokes et al. · 0 citations
Preprint Jul 2026

Inverse RL Helps Align AI by Imitating Humans

It is shown that the recovered reward improves a base policy without a supervised loss and yields further gains when optimized after standard supervised fine-tuning and can be used for contextual alignment, in which a single policy can be tailored to the preferences of different audiences.

Michal Wilinski, Liu Leqi, Chirag Nagpal · 0 citations
Preprint Jul 2026

Learning Latent Reasoning Traces for Scalar Reward Models End-to-End

LatentRM is a reward modeling framework that learns intermediate reasoning traces as discrete latent variables to explicitly maximize the likelihood of downstream scalar rewards through on-policy optimization of the latent reasoning space end-to-end.

Sanwoo Lee, Clive Bai, Hsiu-Yuan Huang et al. · 0 citations
Preprint Aug 2026

StructReward: Efficient Structured Process Rewards for Self-Correcting Multimodal Reasoning

This work introduces StructReward, a compute-efficient framework that provides dense reinforcement signals through structured step-level reward alignment and substantially reduces the computational overhead of multimodal reinforcement learning.

Yifan Li, Ruxi Sun, Tongzhou Zhao · 0 citations
Preprint Jul 2026

Optimizing Visual Generative Models via Distribution-wise Rewards

A novel framework that finetunes generative models using distribution-wise rewards, ensuring better alignment with real-world data distributions is presented, and a subset-replace strategy that efficiently provides reward signals by updating only a small subset of a generated reference set is introduced.

Ruihang Li, Mengde Xu, Shuyang Gu et al. · 0 citations