Aligning Flow-Matching Vision-Language-Action Models via Advantage-Guided Action Preference Optimization
Abstract
Deploying robust manipulation policies on robotic arms with lightweight Vision-Language-Action (VLA) models remains challenging. Teleoperated demonstrations often contain human hesitation and suboptimal corrections, which Supervised Fine-Tuning (SFT) inadvertently mimics rather than learning decisive control. In this paper, we propose the Advantage-Guided Flow-Matching Action Preference Optimization (AFAPO), an online fine-tuning algorithm that iteratively refines a continuous VLA using successful human-in-loop trajectories. First, we train a distributional value function on mixed interaction data to extract advantage estimates. This dynamically assigns binary preference labels to state-action pairs, filtering out teleoperation noise without manual annotation. Second, we introduce a tractable preference objective for continuous flow-matching policies by approximating the intractable log-likelihood ratio via the Evidence Lower Bound (ELBO), which reduces the implicit reward to the difference in L2 denoising errors between the current and reference models. Combined with the Kahneman-Tversky Optimization (KTO), this yields a single-sample preference loss. Validated on a task of placing earplugs into the box using a SO-101 robotic arm, AFAPO achieves 85% success rate which shows a significant improvement and reduces completion time compared to plain SFT and heuristic baselines.