Vision-Language Guided Aerial Manipulation for Autonomous UAV Grasping

Abstract

Unmanned aerial vehicles (UAVs) equipped with manipulators enable aerial interaction tasks such as object pickup and transport in environments that are difficult for ground robots to access. However, reliable autonomous aerial grasping remains challenging due to limited onboard perception, dynamic flight conditions, and the need for precise visual alignment. This paper investigates the use of visionlanguage models (VLMs) to guide UAV-based aerial manipulation through visual reasoning and command generation. We present a system that integrates a downward-facing camera, an onboard flight controller, and an external GPU-based VLM inference module to generate motion commands for objectcentered grasping. Two control strategies are explored: (1) direct prediction of low-level motion commands and (2) highlevel language-mediated control for iterative target alignment. A dataset of aerial manipulation samples is collected in simulation to fine-tune several open-source VLMs using LoRAbased parameter adaptation. The approach is evaluated in both simulation and real-world experiments using multiple object types and environmental conditions. Experimental results demonstrate that the fine-tuned models achieves the highest grasp success rate and improved centering accuracy compared with other evaluated models, highlighting the potential of VLMguided perception and control for flexible aerial manipulation with UAV platforms.