VLM-Augmented Behavior Trees for Autonomous Aerial Object Retrieval under Semantic Uncertainty
Abstract
Autonomous aerial object retrieval requires reliable high-level decision-making particularly when perceptiondriven conditions govern task progression. Conventional Behavior Trees (BTs) are widely used to structure such missions, but their deterministic condition evaluations limit the use of complex semantic perception. Vision-Language Models (VLMs) offer rich semantic reasoning through natural-language queries and therefore appear promising for this purpose; however, their outputs are uncalibrated and inherently uncertain, making direct integration into BTs brittle for safety-critical robotic systems. In this work, we bring these two perspectives together and propose a VLM-Augmented Behavior Tree framework for autonomous aerial manipulation that explicitly accounts for semantic uncertainty. VLM responses are treated as noisy observations of latent task predicates, including target visibility and safe camera transition. A Bayesian belief update mechanism is embedded within phase-specific BT subtrees to enable belief-space decision-making. To model VLM reliability, we introduce both constant and input-conditioned asymmetric confusion models, in which error rates adapt to the image blur measure and the UAV velocity magnitude. Experiments in a simulated aerial object retrieval mission demonstrate that the proposed belief-space reasoning framework improves robustness to semantic uncertainty, increasing success rate, while reducing recovery actions and mission time compared to deterministic and constant-error baselines.