Jan 2025· arXiv.org· Vol abs/2501.05819· 9 citations· 15 references
Computer Science
TL;DR
Simulation results confirm the effectiveness and benefits of DMs in generating neighbor velocity estimates in a four-UAV swarm coordination task using Deep Reinforcement Learning (DRL), and explore the integration of DMs with RL and DT.
Abstract
Uncrewed Aerial Vehicles (UAVs) are increasingly used in modern communication networks. However, challenges in decision-making and digital modeling continue to hinder their rapid development. Reinforcement Learning (RL) algorithms face limitations such as low sample efficiency and limited data versatility, which are further amplified in UAV communications scenarios. Additionally, Digital Twin (DT) modeling presents significant challenges in decision-making and data management. RL models, often integrated into DT frameworks to address these issues, require large amounts of training data to make accurate predictions. Unlike traditional approaches that focus on class boundaries, Diffusion Models (DMs)-a new class of generative AI-learn the underlying probability distribution from training data and can generate reliable new patterns based on this learned distribution. DT and RL have complementary roles in enabling intelligent, data-driven UAV operations. DMs further enhance this synergy by addressing data scarcity, improving modeling accuracy, and generating realistic scenarios, which benefit both DT simulations and RL training. In this paper, we explore the integration of DMs with RL and DT. Simulation results confirm the effectiveness and benefits of DMs in generating neighbor velocity estimates in a four-UAV swarm coordination task using Deep Reinforcement Learning (DRL).
Using Unmanned Aerial Vehicle (UAV) for urban sensing has emerged as a powerful paradigm to monitor the status of the city, e.g., air quality and noise levels, through agile aerial crowdsourcing. Despite this potential, existing UAV-based sensing approaches overlook environmental disturbances like wind that drastically impact drone velocity and energy efficiency. Consequently, directly applying existing methods to this joint delivery and sensing paradigm in dynamic environments faces two severe challenges: (1) scalability bottlenecks as fleet sizes expand; and (2) multi-timescale decision heterogeneity between macro task dispatching and micro velocity control. To tackle these, we formalize the problem as SensUAV and propose a Two TimeScale Reinforcement Learning framework (TSRL). Specifically, TSRL separates decision-making into two cooperative layers. At the macro level, a task-embedding sensing dispatcher handles scalability by separately encoding distinct task features and sequentially evaluating UAV suitability before task selection. At the micro level, a wind-aware velocity controller learns fine-grained velocity scheduling to adapt to dynamic environmental variations. Extensive experiments on real-world datasets demonstrate that TSRL significantly outperforms baselines, achieving average system profit improvements of 20.1% in Hangzhou and 46.6% in Shanghai.
Ouyang Xin, Songxin Lei, Xusen Guo et al.· 0 citations
: Unmanned Aerial Vehicles (UAVs) are finding more and more applications in logistics, surveillance, and other operations at a large scale. However, autonomous navigation in dynamic traffic situations is not an easy task due to limited energy, moving obstacles, and inter-agent interactions. The proposed paper can be discussed as a Generative World Modeling (GWM) framework of risk-focused UAV navigation in the dynamic traffic network. This paper proposes a GWM framework for risk-aware UAV navigation in dynamic traffic networks. The proposed design incorporates three key elements; a generative world model for predicting future environmental conditions, a diffusion-based trajectory-generation component that generates multiple possible paths, and a risk-aware decision-making component that selects trajectories based on energy use, collision avoidance, and mission criteria. The framework is also extended to the case of a multi-UAV swarm, where a coordinated swarm is facilitated by shared representations in the latent space to alleviate potential conflicts. The experimental analysis of real-world-inspired UAV trajectory data indicates that the proposed GWM framework outperforms the classical, reinforcement-based, and conflict-aware baseline approaches across a range of performance metrics, including mission success rate, delivery time, energy cost, safety, and path efficiency. The findings indicate that with risk-sensitive and generative prediction, autonomous UAV missions should be more robust, effective, and secure in uncertain, complex environments.
Alaa M. Momani, D. Alsekait, M. Al-Khasawneh et al.· Computers, Materials & C...· 0 citations
Unmanned Aerial Vehicles (UAVs) have gained widespread attention in diverse applications like military, medical, aerial surveillance and many more. Presently, the problem of limited bandwidth and geographic factors has raised the need for effective and timely data transfer. Training UAVs with reinforcement learning-based algorithms facilitates autonomous decision-making capabilities. In this paper, we proposed an intelligent system for the optimal UAV selection process by evaluating the continuous performance of each UAV. The analyzing factors are based on the real-world factors affecting the quality of signals, such as noise interference, relative motion between source and wave, and transmission power. Based on the systematic conditions observed, the system provides efficient rewards. To promote the selection of the optimal UAV and enhance the learning process, the state information of the UAV is fed into a deep neural network (DQN), which predicts the 'Q-values'. Our system implements a deep Q-learning algorithm, which enhances the agent's performance by systematically learning from its experience. The model operates accurately by selecting the most reliable UAV, thus, enhancing the throughput by optimal power allocation. It outperforms other conventional models in terms of timely data delivery and energy utilization. The system adapts various complex patterns by analyzing the historical and present scenarios. Empowered by this intelligent system, time-critical decision-making can be achieved with minimal energy consumption.
Divyanshu Bhardwaj, Angel Kanjiya, N. Jadav et al.· 2026 IEEE International Work...· 0 citations
This study proposes a novel Deep Reinforcement Learning (DRL)-based resource allocation architecture that dynamically mitigates physical layer impairments in Long Range (LoRa) communication networks established with Unmanned Aerial Vehicles (UAVs) operating at tactical speeds. Traditional Adaptive Data Rate (ADR) algorithms used in LoRaWAN networks misinterpret the Doppler shift under high mobility as path loss, leading to an unwarranted increase in the spreading factor and subsequent communication link failures. In this work, a cross-layer Deep Q-Network (DQN) agent is designed to incorporate UAV velocity into the state space, autonomously selecting the optimal spreading factor and transmission power by predicting frequency shifts at the physical layer. Simulations conducted in a realistic Rayleigh fading channel model demonstrate that the proposed method increases the Packet Delivery Ratio (PDR) to over 85% at high speeds, significantly outperforming conventional algorithms.
Sıtkı Öztürk, İlyas Soyer· Signal Processing and Commun...· 0 citations
Rapid advances in drone technology, combined with the growing congestion of terrestrial transport networks, are driving the emergence of the low-altitude economy. Uncrewed Aerial Vehicles (UAVs) are increasingly deployed for low-altitude economy applications such as urban logistics and transportation, yet their expansion is constrained by the scarcity of spectrum resources. Although Multi-Agent Reinforcement Learning (MARL) offers a promising decentralized approach to improve spectral efficiency of UAVs, existing MARL methods suffer from high training costs, often requiring extensive environmental interactions. To overcome these limitations, we propose a novel Multi-Agent Model-Based reinforcement learning algorithm for decentralized spectrum sharing among UAVs in the low-altitude economy, which we denote as MAMBA-UAV. Adopting a Centralized Training with Decentralized Execution (CTDE) paradigm, MAMBA-UAV equips each UAV with a learned world model that captures compact environmental representations and predicts system dynamics. These world models are then utilized during MARL training to simulate interactions, thereby reducing the reliance on repeated real-environment rollouts. Through comprehensive simulations, we demonstrate that MAMBA-UAV substantially reduces the number of environmental interactions required for UAVs to achieve competitive spectrum-sharing performance, lowering training costs while maintaining high performance.
Tianle Li, Peixi Peng, Qingyu Liu et al.· IEEE Transactions on Cogniti...· 0 citations
Unmanned aerial vehicles play an increasingly important role in the low-altitude domain by collecting and transmitting aerial images. However, the inter-dependency between UAV motion and communication strategies has been largely overlooked. To address this gap, we propose a UAV motion-aware image capturing and communication (MICC) system that dynamically optimizes data offloading by jointly considering scenario variability and communication resource allocation. Specifically, we formulate an MICC optimization problem to maximize transmission accuracy and efficiency by adaptively controlling down-sampling ratios, compression ratios, and transmit power. Considering its non-convex nature, we first develop a geometric programming based algorithm (GP-MICC) to obtain high-fidelity solutions. Recognizing its high computational cost, which hinders real-time deployment, we further propose a fast learning-based optimization algorithm (FLO-MICC). Extensive experiments demonstrate that GP-MICC achieves excellent transmission performance, while FLO-MICC reduces computational time by over 12x with minimal performance loss, making it suitable for dynamic UAV scenarios.
