A reinforcement learning approach using the TD3 algorithm to train autonomous agents to reach arbitrary targets within a parametric, chaotic double-gyre flow is presented and a trade-off in sensor utility is revealed: velocity-aware agents optimize energy efficiency, whereas vorticity sensors provide superior structural mapping and achieve better target proximity.
Abstract
Autonomous robotic navigation in nonstationary time-varying fluid flows remains a fundamental challenge due to partial observability and the unpredictability of realistic environments. While classical optimal control frameworks employed in robotics require unrealistic a-priori global flow knowledge, biological systems are able to navigate successfully by exploiting localized sensory cues. In this work we present a reinforcement learning approach using the TD3 algorithm to train autonomous agents to reach arbitrary targets within a parametric, chaotic double-gyre flow. To investigate optimal sensory mechanisms, we evaluate five bio-inspired observation strategies based on relative position, local velocity or local vorticity measures, and short-term memory variants. Additionally, we analyze the impact of providing agents with explicit global flow parameters. Numerical results demonstrate that an agent that is able to sense and remember a set number of flow velocity measures achieves the highest performance. The experiments reveal a trade-off in sensor utility: velocity-aware agents optimize energy efficiency, whereas vorticity sensors provide superior structural mapping and achieve better target proximity. Incorporating explicit global flow parameters is shown to decrease navigation performance. This behavior suggests that reinforcement learning-based autonomous systems develop more robust and general policies when restricted to implicit flow representations. The presented results offer insights for improving the transition of bio-inspired robotic navigation from simulation to real-world environments.
PEARL employs an actor-adjoint algorithm that leverages automatic differentiation to compute policy gradients over short horizons and adjoint-based sensitivities of future returns approximated via neural networks, significantly reducing the number of environment interactions, while mitigating long-term gradient instabilities.
Matteo Tomasetto, Nicolò Botteghi, Gabriele Bruni et al.· 0 citations
Navigating in unsteady wake flows, such as Kármán vortex streets, presents a formidable challenge for biomimetic autonomous underwater vehicles. Biological fish achieve this by utilizing their lateral line sensory systems to perceive local flow gradients and adopting an energy-efficient swimming pattern known as the Kármán gait. To translate this biological phenomenon into a practical robotics engineering solution, this paper proposes a fully computational framework focusing on the modeling and simulation of a spatio-temporal sensory system to autonomously generate the Kármán gait. To overcome the unrealistic assumption of full-state observability common in existing reinforcement learning studies, we model a multi-point lateral line array coupled with a frame-stacking mechanism. This allows the simulated agent to reconstruct the spatio-temporal topology of the surrounding unsteady flow relying exclusively on local pressure and velocity gradients. The sensory model is integrated with a spatio-temporal perceptual twin delayed deep deterministic policy gradient (STP-TD3) algorithm, which drives a Hopf-oscillator-based central pattern generator. Through rigorous high-fidelity computational fluid dynamics simulations, we quantitatively evaluate the autonomous emergence of the Kármán gait by assessing the agent’s kinematic energy proxy-mapped from joint actuation effort. Results reveal that the agent expends significantly less mechanical effort navigating through the turbulent vortex street compared to swimming in steady water, suggesting the active exploitation of the local wake dynamics. The results theoretically underscore the necessity of distributed STP for biomimetic robots, providing a robust algorithmic blueprint for future physical deployments in complex aquatic environments.
Xinqi Wang, Ming Wang, Xinyan Liu et al.· Bioinspiration & Biomimetics· 0 citations
DRIQN is proposed to integrate Distributionally Robust Optimization (DRO) with implicit quantile networks to optimize worst-case performance under natural environmental conditions and incorporates heterogeneous noise sources and target robustness-critical scenarios.
Zhaofan Zhang, Minghao Yang, Sihong Xie et al.· 0 citations
Repetitive maritime operations can be effectively learned using the Imitation Learning (IL) paradigm, which transfers human expertise directly to Unmanned Surface Vehicle (USV) control systems. Dynamical Systems (DS) are widely used to model non-linear human demonstrations while offering inherent stability guarantees. However, real-world execution under persistent marine perturbations reveals a critical trade-off: standard DS-based IL approaches prioritize global target convergence at the expense of localized trajectory reproduction fidelity. To address this limitation, we present a hybrid learning-control architecture that integrates a DS-based IL reference generator with a neuroadaptive controller. Our approach introduces a control action that drives the USV back to the demonstrated path following exogenous disturbances, enabling dynamic human-like reactive alignment-termed behavioral tracking. The proposed methodology is validated using the Marine Systems Simulator (MSS) toolbox. Simulation results confirm that the framework generalizes complex maneuvering tasks while substantially improving trajectory tracking fidelity under disturbances compared to alternative control strategies.
Yeyson A. Becerra-Mora, José Ángel Acosta· 0 citations
Approaching a target position and holding station in flowing water is a fundamental and critical capability for robotic fish operating in natural aquatic environments. Despite decades of advances in enhancing swimming efficiency and maneuverability, this capability remains underdeveloped, largely owing to the insufficiently characterized, highly nonlinear fluid-structure interactions inherent to freely swimming robotic fish in flows. To bridge this gap, we propose the SWiFT framework, a Swimming With Flow Toolbox that enables the efficient exploration of an egocentric station-holding policy for a body and/or caudal fin (BCF) robotic fish in unknown and turbulent background flows via reinforcement learning (RL). Our SWiFT integrates a free-swimming flow-tank experimental setup with a highly efficient, physically consistent computational fluid dynamics (CFD)-based simulator and a systematic sim-to-real transfer pipeline. The resulting policy achieves substantial improvements over state-of-the-art methods across all metrics, most notably root-mean-square error (RMSE) of distance. Furthermore, we validated that egocentric feedback alone, without any explicit flow sensing, enables station-holding in unknown turbulent flows, closely mirroring the biological phenomenon of rheotaxis. Accordingly, the success of this egocentric station-holding policy not only advances robotic fish control toward real-world deployment, but also highlights SWiFT's promise as a foundation for tackling complex swimming tasks for underwater robots.
Xiaozhu Lin, Xuejiao Huang, Hongru Dai et al.· 0 citations
This paper demonstrates how deep reinforcement learning (DRL) enables adaptive locomotion of snake-like robots in dynamically changing viscous environments, overcoming the inherent performance limitations of classical predefined control methods. The lack of direct onboard sensors for fluid properties necessitates formulating this task as a partially observable Markov decision process. By employing an asymmetric actor-critic framework, a teacher policy trained using privileged information available only in the physics simulator distills its knowledge into a student policy that relies solely on proprioceptive sensor information. Simulation results across a wide range of dynamic viscosity changes ($10^{-7}$ to $10^{-2} m^2/s$) reveal that the DRL agent autonomously acquires non-sinusoidal adaptive gaits. These gaits improve propulsion velocity and transport efficiency, breaking the inherent limits of conventional sinusoidal and kinematic control. The findings establish that implicit environment inference via privileged information distillation is an effective approach to bypass the constraints of classical models under unpredictable fluid dynamics.
T. Kimoto, A. Yamano, Kohei Honda et al.· 0 citations