Queue-Aware Deep Reinforcement Learning for UAV-Assisted Maritime Integrated Sensing and Communication

Abstract

This paper investigates an unmanned aerial vehicle (UAV)-assisted maritime integrated sensing and communication (M-ISAC) system, where a UAV periodically senses sea-surface targets and transmits the collected data to an unmanned surface vehicle (USV) for further processing. In addition to analyzing the estimation information rate and achievable transmission rate in maritime environments, we incorporate data queue stability at the UAV to characterize the inherent performance tradeoff between sensing and communication in dynamic maritime environment. Our goal is to minimize the long-term average system energy consumption by jointly optimizing the UAV trajectory, beamforming, target scheduling, and time partitioning. In order to efficiently solve the formulated problem, we propose a queue-aware deep reinforcement learning framework, namely, Lyapunov-guided soft actor-critic (L-SAC), which transforms the problem into an instantaneous mixed-integer nonlinear programming (MINLP) problem in each time slot by Lyapunov optimization, and subsequently solves it using reward-shaped SAC algorithm. Simulation results demonstrate the effectiveness of our proposed framework and reveal the impact of data queue dynamics on the sensing-communication trade-off.