Joint 3D Trajectory and Power Optimization for UAV Swarms in Cell-Free Massive MIMO Networks: A CTDE-MAPPO Framework for Sensing-Aware Precision Agriculture
A multi-agent deep reinforcement learning (MADRL) methodology based on the Multi-Agent Proximal Policy Optimization (MAPPO) approach, which simultaneously achieves high field coverage completeness, robust communication energy efficiency, and a high depot-return rate under hard battery constraints without any inter-UAV communication overhead at execution time.
Abstract
The integration of Unmanned Aerial Vehicle (UAV) swarms with Cell-Free massive Multiple Input Multiple Output (CF-mMIMO) networks offers promising prospects for large-scale crop monitoring in precision agriculture. CF-mMIMO provides macro-diversity and uniform channel quality across large agricultural fields. However, practical deployment demands jointly optimizing 3D trajectories and transmit power to maximize energy efficiency and field coverage simultaneously. This is challenging due to the limited battery capacity, mandatory return-to-depot constraints, and collision avoidance requirements. In this paper, we introduce a joint sensing–communication utility function that captures the trade-off between energy efficiency and field coverage completeness. To provide a scalable and distributed solution for rotary-wing UAV swarms, we develop a multi-agent deep reinforcement learning (MADRL) methodology based on the Multi-Agent Proximal Policy Optimization (MAPPO) approach. We adopt the Centralized Training with Decentralized Execution (CTDE) strategy, in which a CF-mMIMO central processing unit (CPU) serves as a global critic during training. At execution time, each UAV independently runs a lightweight local policy that adapts its trajectory and transmit power in real time based on battery state and air-to-ground channel variations. Simulation results reveal that the proposed MAPPO-CTDE approach outperforms existing benchmarks. Unlike prior methods that require instantaneous global CSI or neglect the sensing–communication coupling, the proposed approach simultaneously achieves high field coverage completeness, robust communication energy efficiency, and a high depot-return rate under hard battery constraints without any inter-UAV communication overhead at execution time.
: The emergence of Unmanned Aerial Vehicle (UAV)-enabled Wireless Energy Transfer (WET) and Simultaneous Wireless Information and Power Transfer (SWIPT) technology provide a promising solution to overcome the energy sustainability limitations of traditional harvesting-reliant sensor networks. However, in large-scale Battery-free SWIPT-enabled Sensor Networks (BSSN) characterized by sparse node distribution and heterogeneous energy consumption and harvesting rates, employing a single UAV for energy replenishment often suffers from insufficient operation continuity and low charging efficiency. To overcome these challenges, a Multi-UAV Collaborative Energy Charging for BSSN Based on Multi-Agent Deep Deterministic Policy Gradient (MCEC-MADDPG) is proposed in this paper. Specifically, we construct a collaborative one-to-one precision energy supply model where UAVs hover directly above specific nodes to achieve power transmission without complex beamforming requirements. To achieve collaborative scheduling among multiple UAVs in wide-area dynamic environments, the energy replenishment problem is first formulated as a Partially Observable Markov Decision Process (POMDP). Subsequently, the Centralized Training with Decentralized Execution (CTDE) architecture of the MADDPG algorithm is leveraged to solve this POMDP, which effectively tackles the non-stationarity challenge inherent in multi-agent environments. Simulation results demonstrate that MCEC-MADDPG exhibits superior performance in terms of convergence speed and stability. It enables the adaptive emergence of spatial-division collaborative strategies, significantly enhances the average residual energy of the network, and elevates the node survival rate to nearly 90%. Compared with Deep Deterministic Policy Gradient (DDPG), the traditional static Partition-Greedy method, the heuristic K-Means algorithm and the dynamic Two-Layer task allocation strategy, the proposed approach demonstrates substantial advantages.
Xiangyi Le, Deyu Lin, Yufei Zhao et al.· Computers, Materials & C...· 0 citations
Unmanned aerial vehicle (UAV) communications are a promising enabler for 6G networks, offering flexible deployment and strong line-of-sight channel conditions. Effective UAV operation requires jointly optimizing trajectory and user scheduling to balance throughput and information freshness. This paper proposes a proximal policy optimization (PPO)-based deep reinforcement learning (DRL) framework that controls UAV movement and user scheduling together via a joint MultiDiscrete action space. We formulate a Markov decision process for a 8-user, $1000 \times 1000 \mathrm{~m}^{2}$ service area with a 3GPP TR 36.777-compliant channel model, where the agent selects both its next position and which user to serve at each time slot. The proposed PPO policy achieves 85.75 Mbps mean throughput, a 24.4% improvement over the AoI-greedy baseline, while reducing mean AoI by 87.7% compared to the throughput-greedy baseline, reaching a Pareto-optimal trade-off between the two competing objectives. An ablation study over the AoI penalty weight confirms a clear throughput-AoI trade-off, validating the joint design.
Quang Tuan Do, Tung Son Do, Thanh Phung Truong et al.· International Conference on...· 0 citations
This paper investigates the problem of cooperative multiple unmanned aerial vehicles (UAVs) data collection for Internet of Things (IoT) networks in dense urban environments. Unlike existing studies that predominantly rely on idealized spatial models and average-based probabilistic channel models, this work explicitly accounts for realistic 3-D building distributions and deterministically models ground-to-air (G2A) channel blockages. We formulate a joint optimization problem to minimize the total task completion time, subject to stringent system throughput, flight dynamics, and energy constraints. To tackle the highly coupled challenges of node scheduling and trajectory planning, we propose a lightweight two-stage heuristic strategy for dynamic access control, along with a multi-agent reinforcement learning for trajectory planning. Crucially, to overcome the severe sparse-reward bottleneck inherent in complex 3-D obstacle avoidance, we introduce a Pheromone-based Reward Shaping (PRS) mechanism. By mathematically integrating the UAV’s kinematic state with deterministic environmental feedback, PRS effectively transforms the sparse-reward navigation challenge into a dense and smooth gradient, thereby profoundly accelerating policy convergence. Extensive simulations demonstrate that the proposed MATD3-PRS framework significantly outperforms representative baselines, achieving superior performance in task completion time, flight trajectory efficiency, and overall energy saving.
Haitao Chen, Xinfeng Deng, Zhe Wang et al.· IEEE Transactions on Cogniti...· 0 citations
This study investigates the optimization of three-dimensional (3D) trajectory planning and resource allocation in unmanned aerial vehicle (UAV)-enabled wireless networks with no-fly zones (NFZs) using a deep learning framework. The objective is to maximize the minimum average spectral efficiency (SE) among mobile users served by multiple UAVs while addressing key challenges, including interference from concurrent UAV transmissions, collision avoidance, and NFZ constraints. A realistic probabilistic channel model is considered, where the likelihood of a line-of-sight (LoS) condition is modeled as a function of the elevation angle in the air-to-ground (A2G) link. To solve the formulated optimization problem, a novel deep learning framework with specialized deep neural network (DNN) structures is developed. This framework jointly optimizes 3D UAV trajectory planning and resource allocation, employing an unsupervised learning-based training approach that eliminates the need for labeled data. Performance evaluations demonstrate that the proposed scheme effectively accounts for the probabilistic channel model and co-channel interference while accounting for collision avoidance and NFZ-related constraints. Moreover, it outperforms baseline methods by achieving a higher minimum average SE with real-time computational efficiency, making it practical for UAV-assisted wireless networks.
Unmanned Aerial Vehicles (UAVs) are promising relay platforms due to their flexible deployment and high probability of line-of-sight (LoS) connectivity. This paper compares three deep reinforcement learning (DRL) algorithms-Proximal Policy Optimization (PPO), Soft Actor-Critic (SAC), and Recurrent PPO with LSTM memory-for joint UAV trajectory and energy optimization in UAV based relay systems. The problem formulated is a non-convex optimization problem that minimizes UAV propulsion energy while satisfying Quality of Service (QoS) and mobility constraints under realistic 3GPP channel conditions. Simulation results show that all methods achieve over 99% QoS satisfaction. SAC exhibits the fastest convergence, whereas the proposed Recurrent PPO achieves the lowest energy consumption (44.72 kJ), reducing energy usage by 5.1% compared with PPO. These results highlight the trade-off between convergence speed and energy efficiency in DRL-based UAV relay optimization.
Aniket Subbanwar, Ojas Joshi, Amit Agarwal· International Conference on...· 0 citations
Unmanned aerial vehicles (UAVs) have emerged as a key enabler of next-generation Internet of Things (IoT) ecosystems, offering flexible aerial relaying to extend connectivity across dynamic vehicular ad hoc networks (VANETs) in smart city environments. However, conventional centralized approaches for UAV trajectory planning require continuous global network state aggregation, making them impractical under bandwidth and energy constraints typical of dense urban deployments. In this article, we present TRUAV, a distributed multi-agent reinforcement learning framework based on independent tabular Q-learning for joint UAV trajectory planning and routing enhancement in UAV-aided VANETs. Each UAV is equipped with a local Q-learning agent that operates purely on locally observable information, including vehicle density, packet queue states, and neighbor UAV positions, thereby eliminating the need for global state exchange. A potential-game-inspired reward design encourages spatial diversity and routing-aware UAV positioning among interacting agents while accounting for energy consumption. Numerical simulations over a large urban area with 200 mobile vehicles show that the proposed TRUAV framework achieves network coverage and packet delivery ratios comparable to centralized deep reinforcement learning methods, while also improving relay delay and energy efficiency. Finally, we discuss emerging challenges and future research directions for distributed multi-agent UAV-assisted IoT systems.
Muhammad Umar Farooq Qaisar, Lin Zhang, Zhen Chen et al.· 0 citations