This work dissects advantage-guided VLA post-training through a controlled empirical study that separates these design choices while accounting for their distinct estimands, and develops stage-specific offline evaluation methods to screen alternative choices efficiently, without requiring extensive real-robot policy evaluations.
Abstract
Advantage-guided reinforcement learning provides a practical way to post-train vision-language-action (VLA) policies using limited robot data. However, its performance depends on several coupled choices, including how critic-derived advantages are constructed, calibrated, and used for policy training. Existing recipes often combine these choices into a single end-to-end procedure, making their individual effects difficult to identify. In this work, we dissect advantage-guided VLA post-training through a controlled empirical study that separates these design choices while accounting for their distinct estimands. We develop stage-specific offline evaluation methods to screen alternative choices efficiently, without requiring extensive real-robot policy evaluations for every possible combination. The staged evaluation identifies a modular recipe that combines temporal-difference advantage construction, group-wise calibration, and continuous advantage weighting. Across four real-world bimanual tasks, the resulting recipe improves mean task progress and success over the SFT initialization by 0.42 and 0.63, respectively. Moreover, the proposed evaluation diagnostics show an overall alignment with downstream real-world performance, supporting their use for interpreting empirical outcomes and selecting advantage-guided post-training designs in practice.
A constrained multi-objective post-training method for this setting was developed and validated, and it was verified that the method’s internal constraint-enforcement mechanism behaves as intended, applying negligible pressure while the requirement is easy to satisfy and increasing sharply as it becomes difficult to me...
C. Hill, Zain Ulabedeen Farhat, Yue Wang· 0 citations
WISE (World-model-guided Imagination Scheduling for Efficient Post-training of Vision-Language-Action Models), a unified framework that coordinates when and how world-model imagination is used during policy refinement, is introduced.
Chen-Hao Zhang, Han-Yu Zhao, Hang Cheng et al.· 1 citation
A reliable TTT framework for VLA policies (VANE), where candidate updates are isolated from the live policy, evaluated on subsequent observations, and committed only when supported by future evidence, making adaptation selective and reversible.
H. Ji, Guo-Yang Xia, Luo-Yang Sun et al.· 1 citation
Large-scale vision-language-action (VLA) models provide powerful priors for robot manipulation, yet adapting them to a specific deployment remains challenging. Supervised fine-tuning (SFT) on task-specific demonstrations provides a step toward deployment, but faces two persistent limitations: static data provide limite...
FIND is introduced, an agentic real-world RL framework that closes the loop between scene understanding, weakness-aware practice, self-evaluation, and policy improvement in a persistent workspace and reframes autonomous practice as a scene-conditioned, performance-aware task-selection problem.
Temporal GRPO addresses the problem of trajectory-level credit aliasing in post-train VLA policies by constructing detectable task stages, aligning each rollout with stage-specific action intervals, and comparing only rollouts that have entered the same stage.
Yao Zhou, Hang Gao, Fengge Wu et al.· 2 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.