This work proposes ARIES-Mission2, a zero-shot Vision-Language-Action (VLA) framework that decouples visual-semantic perception from physical route optimization in low-altitude Unmanned Aerial Vehicle (UAV) mission generation and indicates that the TSP module maintains lower growth in computation time as the number of targets increases.
Abstract
Multimodal Large Language Models (MLLMs) have shown strong semantic understanding capabilities, but their direct use in low-altitude Unmanned Aerial Vehicle (UAV) mission generation remains limited by weak spatial optimization and inefficient route planning. To address this issue, we propose ARIES-Mission2, a zero-shot Vision-Language-Action (VLA) framework that decouples visual-semantic perception from physical route optimization. Given natural-language instructions and satellite imagery, ARIES-Mission2 first uses DeepSeek-V3 for task parsing and Molmo-7B for zero-shot target grounding, and then converts detected pixel locations into GPS waypoints through geospatial interpolation. To reduce the redundant backtracking caused by raw VLM-generated visiting orders, the back end formulates multi-target UAV traversal as a Traveling Salesperson Problem (TSP) and compares four candidate routes, including the raw VLM order and the routes optimized by PSO, GPSO, and IPSO. The minimum-cost closed-loop route is then selected for mission generation. Experiments on the UAV-VLPA-nano-30 benchmark show that ARIES-Mission2 achieves a total flight distance of 62.43 km, reducing the route length by 21.6% compared with the unoptimized VLA baseline (79.66 km) and by 9.5% compared with manual human planning (69.00 km). The complete 30-task workflow takes 575.40 s, averaging 19.18 s per task, which is approximately 3.6 times faster than human expert planning. Component-level timing shows that VLM inference dominates the runtime with 19.02 s per task, while the TSP solver requires only 0.16 s per task. Scalability analysis further indicates that the TSP module maintains lower growth in computation time as the number of targets increases.
Vision-Language Navigation (VLN) enables UAV autonomous navigation in unknown environments by mapping language instructions to real-time visual inputs. Compared with GPS-dependent or pre-programmed navigation, VLN supports intuitive human-machine interaction and stronger environmental adaptability, requiring tight integration of high-level semantic reasoning and low-latency flight control.Existing methods suffer from structural misalignment between global multimodal understanding and sequential action generation, causing jittery trajectories and severe decision latency for long-horizon aerial navigation. To solve this issue, we propose FSD-VLN, a fast-slow dual-system architecture disentangling semantic reasoning and low-latency flight command generation.The framework has two asynchronous branches: a slow stream extracting stable semantic priors from pre-trained vision-language models, and a Diffusion Transformer (DiT) fast stream modeling cross-temporal action distributions to produce consistent flight outputs. We further introduce a time-aware adaptive optimizer to stabilize long-sequence training and reduce gradient oscillation.Large-scale low-altitude simulation experiments show FSD-VLN achieves up to 2X higher navigation success rates on unseen scenes than SOTA methods, while cutting single-action inference delay and total task runtime by over 50%. Our work validates the benefit of decoupled semantic-control modeling and provides a practical paradigm for long-horizon aerial VLN.
Xueke Zhu, Qingyan Meng, Liutao Yu et al.· 1 citation
UAV vision-language navigation (UAV-VLN) focuses on enabling an aerial agent to follow natural-language instructions in open 3D environments from egocentric visual observations. Current approaches suffer from three coupled issues: weak grounding of instruction-relevant landmarks in visual observations, insufficient exploitation of long-horizon history, and unstable decisions under local traps or repeated exploration. To address these issues, we propose a unified semantic-to-decision framework. First, we present an instruction-grounded semantic enhancement module that injects object-level semantics and relative spatial cues into the current observation state. Subsequently, we develop a relevance-aware dynamic temporal aggregation strategy that reweights the full history buffer while converting a few high-relevance frames into structured landmark prompts for the decoder. Finally, we devise a topology-aware decision method that combines local-optimum cognition with group-relative policy optimization under progress, goal, semantic, and path-compliance rewards. Experiments on the widely used AerialVLN and OpenFly benchmarks clearly demonstrate that our method achieves state-of-the-art performance.
Existing methods in Autonomous Valet Parking (AVP) typically rely on pre-built maps, which severely restricts their scalability to unseen environments and open-vocabulary targets. Inspired by the application of Vision-Language Models (VLMs) in Vision-Language Navigation (VLN) tasks, we propose VLN-AVP, a zero-shot navigation framework for AVP tasks. By combining the precise spatial perception of a Bird's-Eye-View (BEV) model with the general intelligence of VLMs, our framework 1) eliminates the dependency on pre-built maps, 2) interprets semantic environmental contexts in parking scenarios, and 3) enables intuitive navigation following natural language instructions. Specifically, we introduce a hybrid memory system: a short-term perception memory tracks semantic visual cues to address the limitations of VLM's single-frame reasoning in existing methods, while a long-term topological memory facilitates stable policy learning from past experiences. To bridge the gap in existing benchmarks, we also present the VLN-AVP dataset and benchmark. Featuring 10 high-fidelity parking scenes and over 1,000 navigation episodes, it has the largest number of garage scenes to date and is the first VLN benchmark for underground parking. Extensive experiments demonstrate that in simulation, our method achieves an over 25% improvement in success rate compared to VLN methods and an over 15% improvement compared to other autonomous driving methods. Furthermore, it attains a leading success rate in real-world vehicle experiments, proving its practical feasibility.
Yijiang Li, Xiangru Mu, Changze Li et al.· 0 citations
Deploying vision-language models (VLMs) for safety-critical spatial reasoning on resource-constrained autonomous driving platforms requires both compact model size and reliable metric grounding. We present MoRAL (Multimodal Reasoning for Autonomous Language Models), a two-stage fine-tuning pipeline that teaches Cosmos-Reason2-2B to first read a physics-encoded Bird's Eye View (BEV) representation and then reason over it for driving decisions. The BEV image encodes LiDAR metric distance as color bands, object class as cluster morphology, and radar Doppler velocity as directional wedge overlays, externalizing spatial perception into the input image so that no learned 3D backbone is required at inference. Stage 1 fine-tunes the vision encoder on 60,000 grounding records; zero-shot baselines produce no parseable BEV outputs, confirming the vocabulary requires explicit training. Stage 2 fine-tunes the full model (52M parameters, 2.4% of total) on 57,696 chain-of-thought records generated by Cosmos-Reason2-8B as teacher, spanning eight driving question types. On 2,304 held-out nuScenes frames evaluated by Gemma 4 (31B) calibrated against human review, MoRAL wins seven of eight question types over a zero-shot 8B baseline despite using four times fewer parameters, with the largest margins on question types requiring structured multi-step physics reasoning. Emergency braking recall improves from 10.8% to 47.8%, output degeneration falls from 94.1% to 20.8%, and the full pipeline fits a consumer 8 GB GPU at 42 tok/s without quantization. These results establish a reproducible foundation for compact, physics-grounded VLM reasoning on mobile edge platforms.
Ambarish Govindarajulu Kaliamurthi, Kai Liu· 0 citations
Unmanned aerial vehicles (UAVs) are used for a wide range of tasks, many of which are autonomous. Since UAVs operate in three-dimensional space, visual data presents a rich source of input for them. Meanwhile, the rapid development of artificial intelligence has given rise to vision-language models (VLMs). These models not only recognize objects in a scene, as traditional computer vision techniques do, but also demonstrate a strong understanding of scene context, enabled by large language models (LLMs) serving as one of their core components. However, applications as specialized as aerial robotics typically require custom-trained models, which demand significant computational resources, large datasets, and expensive hardware. Motivated by these limitations, we investigate whether a state-of-the-art, general-purpose VLM can be leveraged for UAV-relevant tasks through carefully designed prompts, without additional training. We evaluate the VLM’s understanding of aerial scenes through the task of image captioning, using visual question answering that targets spatial information within the scene. Furthermore, we utilize VLMs to automatically generate metadata for aerial datasets, enabling more complex future tasks. Results show that VLMs can generate accurate metadata with high agreement with human-annotated data even without retraining. They also demonstrate that careful prompt engineering enhances the model’s ability to reason about and attend to spatial information, highlighting its potential in aerial robotics applications.
Elena Wachtler, Antonella Barisic Kulas, Tamara Petrović et al.· International Conference on...· 0 citations
Coordinating multiple unmanned aerial vehicles (UAVs) for cooperative missions requires agents that perceive their environment, reason about objectives, and generate joint actions. Vision–language–action (VLA) models unify these capabilities but lack a principled source of multi-agent training data and suffer from a training–inference discrepancy in closed-loop control. We propose MiniUAV-VLA, a compact centralized VLA controller for simulated multi-UAV search-and-elimination based on multi-agent reinforcement learning (MARL) expert distillation. A QMIX expert policy achieving 100% mission success generates multimodal demonstrations pairing rendered tactical map images with structured textual state prompts. A 158 M-parameter VLA model with approximately 65 M trainable parameters in the MiniMind-3V backbone and vision projection is fine-tuned with a multi-agent discrete action head that jointly predicts actions for all UAVs in a single forward pass. We identify a training–inference feature mismatch in behavior cloning and address it via prompt-end action pooling, which extracts action-relevant hidden states at the user–prompt boundary rather than after the generated response. In closed-loop evaluation with four drones and six mobile targets averaged over five evaluation seeds, MiniUAV-VLA reaches 74.4 ± 4.6% mission success against 9.4 ± 2.1% for a random policy and 16.2 ± 3.2% for an observation-limited greedy baseline. Across five independent training runs, prompt-end action pooling improves mean closed-loop success from 40.6% to 76.2% over the last-token alternative. These results support MARL expert distillation as a data-efficient route to compact multi-agent VLA control in this simulated setting.
Hongwei Han, Guanghong Gong, Ni Li· Drones· 0 citations