Skip to content

Dissecting Advantage-Guided Post-Training for Vision-Language-Action Policies

Sep 2026 · arXiv.org · Vol abs/2609.28161 · 0 citations · 43 references
Computer Science

TL;DR

This work dissects advantage-guided VLA post-training through a controlled empirical study that separates these design choices while accounting for their distinct estimands, and develops stage-specific offline evaluation methods to screen alternative choices efficiently, without requiring extensive real-robot policy evaluations.

Abstract

Advantage-guided reinforcement learning provides a practical way to post-train vision-language-action (VLA) policies using limited robot data. However, its performance depends on several coupled choices, including how critic-derived advantages are constructed, calibrated, and used for policy training. Existing recipes often combine these choices into a single end-to-end procedure, making their individual effects difficult to identify. In this work, we dissect advantage-guided VLA post-training through a controlled empirical study that separates these design choices while accounting for their distinct estimands. We develop stage-specific offline evaluation methods to screen alternative choices efficiently, without requiring extensive real-robot policy evaluations for every possible combination. The staged evaluation identifies a modular recipe that combines temporal-difference advantage construction, group-wise calibration, and continuous advantage weighting. Across four real-world bimanual tasks, the resulting recipe improves mean task progress and success over the SFT initialization by 0.42 and 0.63, respectively. Moreover, the proposed evaluation diagnostics show an overall alignment with downstream real-world performance, supporting their use for interpreting empirical outcomes and selecting advantage-guided post-training designs in practice.

View source

Similar papers

Constrained Multi-Objective Post-Training for Vision-Language-Action Models

A constrained multi-objective post-training method for this setting was developed and validated, and it was verified that the method’s internal constraint-enforcement mechanism behaves as intended, applying negligible pressure while the requirement is easy to satisfy and increasing sharply as it becomes difficult to me...

C. Hill, Zain Ulabedeen Farhat, Yue Wang · 0 citations

HIL-UMI: Bringing Human-in-the-Loop Post-Training of Vision-Language-Action Models to Universal Manipulation Interface

Large-scale vision-language-action (VLA) models provide powerful priors for robot manipulation, yet adapting them to a specific deployment remains challenging. Supervised fine-tuning (SFT) on task-specific demonstrations provides a step toward deployment, but faces two persistent limitations: static data provide limite...

Zimu Han, Yi-Ming Zeng, Ji-Yao Zhang et al. · 0 citations
Sep 2026

Find Something You Can't Do: Agentic Real-World Reinforcement Learning for Self-Improving VLA Models

FIND is introduced, an agentic real-world RL framework that closes the loop between scene understanding, weakness-aware practice, self-evaluation, and policy improvement in a persistent workspace and reframes autonomous practice as a scene-conditioned, performance-aware task-selection problem.

Yuan Fang, Ze-Chu Li, Hao-Lei Tong et al. · 3 citations · ⚡1
Preprint Aug 2026

Temporal GRPO: Beyond Trajectory-Level Credit in Vision-Language-Action Reinforcement Learning

Temporal GRPO addresses the problem of trajectory-level credit aliasing in post-train VLA policies by constructing detectable task stages, aligning each rollout with stage-specific action intervals, and comparing only rollouts that have entered the same stage.

Yao Zhou, Hang Gao, Fengge Wu et al. · 2 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.