Jul 2026· Problems of Information Technology· 0 citations
TL;DR
An efficient way to resolve the curse of dimensionality, improve obstacle avoidance and cooperative formation control of UAVs was found and shows great prospects of practical application in such domains as military operations, search and rescue missions, transport automation and disaster management.
Abstract
This work considers the issue of multi-agent coordination path planning for multiple UAVs with
different properties working in a dynamic and uncertain environment. In order to resolve the
stated problem, an algorithm based on multi-agent deep reinforcement learning called MultiAgent Soft Actor-Critic (MASAC) is suggested. An original simulation environment is
developed to simulate a scenario of UAVs navigation towards certain goals, taking into account
their kinematics constraints, obstacle avoidance and UAVs heterogeneity. As a consequence, the
task is modeled as partially observable Markov decision process (POMDP). Performance
assessment includes such metrics as task completion ratio, formation, flight efficiency and
energy costs. The suggested framework for MASAC involves centralized training but
decentralized execution which allows agents to coordinate actions and solve POMDP by means
of local perception during testing time. As a result of experimental evaluation, an efficient way
to resolve the curse of dimensionality, improve obstacle avoidance and cooperative formation
control of UAVs was found. Such an approach shows great prospects of practical application in
such domains as military operations, search and rescue missions, transport automation and
disaster management.
A multi-agent guided soft actor–critic (MAGSAC) deep reinforcement learning algorithm to enable multiple UAVs to simultaneously arrive at multiple constant-velocity moving targets and outperforms existing mainstream algorithms in synchronization success rate, temporal synchronization accuracy, and safety.
Shuanli Jia, Naiming Qi, Zheng Li et al.· Drones· 0 citations
Coordinating heterogeneous aerial and ground vehicles with fundamentally different dynamic time-scales is a longstanding challenge in multi-agent robotics. This paper proposes a distributed reinforcement learning (DRL) framework for cooperative Unmanned Aerial Vehicle (UAV) and Unmanned Ground Vehicle (UGV) formation control that explicitly accounts for multirate dynamics: UAVs execute high-frequency control at 10 Hz while UGVs operate at 2 Hz. Each agent maintains a local actor-critic network trained via a variant of MultiAgent Proximal Policy Optimisation (MAPPO) augmented with a consensus-based communication graph and a multirate synchronisation module that bridges the temporal mismatch between the two vehicle classes. A shaped reward formulation penalises formation deviation, inter-agent collisions and communication dropout simultaneously. Simulation experiments In a physicsaccurate Gazebo/ROS2 environment with two UAV leaders, two UAV followers, and two UGVs demonstrate a task success rate of 96.4%, a formation accuracy of 97.1%, and a collision rate of only 0.8%, outperforming MADDPG, MAPPO (singlerate), centralised DDPG and PID baselines by margins of up to 24.9 percentage points. The framework is further validated in a disaster-response scenario with a dynamic obstacle fields confirming its suitability for real-world heterogeneous multirobot missions.
Sankara Rao, P. Palpandi, G. Beulah et al.· International Conference on...· 0 citations
Reinforcement learning techniques have been widely applied to multi-UAV cooperative path-planning tasks. However, existing multi-agent reinforcement learning methods are still affected by environmental non-stationarity, cooperation difficulties among agents, and low utilization efficiency of experience samples in complex obstacle environments. These issues often lead to slow convergence and unstable training performance. To address these problems, an Improved Experience Replay Multi-Agent Deep Deterministic Policy Gradient (IER-MADDPG) algorithm is proposed for multi-UAV cooperative path planning. First, a cooperative path-planning model is established under the Centralized Training Distributed Execution framework. Second, a dual-layer replay buffer structure consisting of a global replay buffer and a local replay buffer is designed to preserve both global cooperative information and individual experience. Third, a fusion experience sampling mechanism is introduced by combining prioritized experience replay and random uniform sampling to improve sample utilization efficiency and training stability. Finally, training experiments were conducted in environments with different obstacle configurations to evaluate the proposed method. Experimental results demonstrate that IER-MADDPG outperforms other comparison algorithms in terms of convergence speed, training stability, and path-planning performance.
Long Wen, Hui Tan, Yuxi Liu et al.· Italian National Conference...· 0 citations
A cooperative guidance law based on the experience-guided multi-agent proximal policy optimization (E-MAPPO) algorithm is proposed for multiple unmanned aerial vehicles (UAVs) to track dynamic points of interest in civilian applications, such as collaborative search and rescue and environmental monitoring. In multi-UAV cooperative tracking, accurate arrival-time coordination is important for improving collaborative task execution, but it remains challenging because of continuous action spaces, target maneuvering, uncertain time-to-go estimation, and inefficient exploration in multi-agent reinforcement learning. Specifically, a multi-UAV cooperative guidance environment is formulated, and the problem is modeled as a Markov decision process. To address the challenges of large action spaces and poor convergence in multi-agent reinforcement learning, an experience-guided MAPPO framework is introduced to enhance training efficiency and policy stability. Different from standard MAPPO, the proposed E-MAPPO introduces proportional-navigation-guided experience only during the early training stage to guide exploration, while the final policy is still optimized through the MAPPO objective. Subsequently, a composite reward function is designed by integrating distance-based heuristic terms with auxiliary guidance signals, thereby improving exploration efficiency and facilitating coordinated rendezvous and tracking of dynamic references. Comparative simulations with cooperative proportional navigation guidance (CPNG), sliding mode control (SMC), and standard MAPPO are conducted under different target motion scenarios. The results show that E-MAPPO reduces the average convergence step by 17.07% compared with MAPPO. In the straight-moving target scenario, E-MAPPO reduces the cooperative time error by 55.10% compared with CPNG and by 8.33% compared with MAPPO. In the S-type maneuvering target scenario, E-MAPPO reduces the cooperative time error by 55.81% compared with CPNG and by 9.52% compared with MAPPO. Monte Carlo experiments further verify its effectiveness and robustness. Additional robustness tests under Gaussian measurement noise, observation bias, and communication delay show that the proposed method maintains acceptable tracking accuracy and cooperative timing performance under different uncertainty conditions. In addition, the results indicate that the proposed method generalizes well to different types of maneuvering targets.
Hao Xiong, Minghu Tan, Xiaoyu Liu et al.· Drones· 0 citations
A distributed real-time trajectory-planning method that integrates a distributed model predictive control framework with an adaptive Gaussian collocation strategy (DA-GCMPC) was developed, which achieves lower computation time and better trajectory quality metrics under the tested simulation settings.
Yang Zhao, Mingying Huo, Naiming Qi et al.· Drones· 0 citations
Deploying Uncrewed Aerial Vehicles (UAVs) for dynamic target search in disaster response scenarios can reduce losses. This paper investigates multi-UAV cooperative trajectory planning for dynamic target search in a three-dimensional environment with static obstacles, aiming to maximize the number of searched targets and minimize the average uncertainty of the search area, while ensuring collision avoidance between UAVs and obstacles. Existing Multi-Agent Reinforcement Learning (MARL) based methods face the sparse reward problem in dynamic target search, which hinders planning feasible multi-UAV trajectories. Notably, Large Language Models (LLMs), with extensive pre-trained knowledge and powerful semantic reasoning capabilities, exhibit potential for designing high-quality reward functions to alleviate the sparse reward problem. Therefore, we propose an LLM-guided Multi-Agent Proximal Policy Optimization (LLM-MAPPO) algorithm, which leverages LLMs’ reasoning capabilities to guide MARL policy learning and plans multi-UAV trajectories for efficient dynamic target search. Specifically, we design an offline LLM reward shaping scheme that generates dense reward signals to mitigate the sparse reward problem. Moreover, we propose a dual-mode pheromone-based search mechanism to guide UAVs to respond promptly to changes in target positions. Experimental results demonstrate that LLM-MAPPO significantly outperforms compared algorithms in terms of the number of searched targets and average area uncertainty, while successfully avoiding collisions. In particular, LLM-MAPPO reduces the target search time by 71.4%.
Yifei Liu, Xiaoshuai Li, Xia Jiang et al.· IEEE Transactions on Cogniti...· 0 citations