reinforcement learning

Design and development of an endoscopic robotic system and deep reinforcement learning path planning algorithm for fine-needle biopsy of liver lesions

Abstract This paper presents the design, development and evaluation of a novel robotic platform for endoscopic ultrasound-guided fine-needle biopsy of liver lesions. The system combines a four degrees-of-freedom (DoF) two-segment tendon-driven continuum robot (TDCR) endoscope with a two DoF superelastic nickel–titanium bevel-tip steerable needle. Needle path planning is achieved using NeedleNav, a soft actor–critic (SAC) deep reinforcement learning (DRL) model that generates collision-free trajectories to deep-seated lesions. This represents one of the first integrated systems combining a TDCR, steerable needle and DRL-based navigation, and the first application of a SAC to liver lesion targeting. Evaluation of the TDCR through tip tracking of circular, diamond-shaped and arc trajectories demonstrated a mean absolute error (MAE) of 13.19 mm. NeedleNav converged to obstacle avoidance trajectories in $$\sim $$ ∼ 2500 training episodes. Needle curvature was augmented by hand-fabricating notches on its distal section. Two needles with a 3-cm and 8-cm notched section were evaluated in a gelatine liver phantom, achieving an MAE of 21.78 mm and 14.86 mm, respectively, for obstacle avoidance trajectories. The system demonstrated observable path deflection compared to obstacle-free trajectories for the same targets. Together, these findings suggest the feasibility of our proposed solution, expanding the reach of endoscopic needle interventions to deep-seated lesions in the right lobe.

Trustworthy Reinforcement Learning for AI-Driven Urban Decision-Making: Sustainable Dynamic Pricing and Resource Optimization for Smart City Operations

Rapid urbanization, increasing demand variability, and digitalisation of urban infrastructure require intelligent decision-support systems that can improve sustainability, resilience, and operational efficiency in smart city operations. Artificial intelligence (AI) and data-driven optimization offer strong potential for adaptive urban services, including dynamic pricing, demand response, resource allocation, and energy-aware management. However, many reinforcement learning applications still focus mainly on short-term performance while giving limited attention to transparency, fairness, stability, and accountability. This study proposes a trustworthy reinforcement learning framework for AI-driven urban decision-making, using sustainable dynamic pricing and resource optimization as mechanisms for adaptive and responsible decision-making. A custom reinforcement learning environment was developed using historical e-commerce transactional data as a methodological proxy to simulate interactions among demand, resource or inventory availability, service categories, price elasticity, and changing market conditions. Three reinforcement learning algorithms, namely Deep Q-Network, Proximal Policy Optimization, and Advantage Actor–Critic, were evaluated under comparable experimental conditions. Performance was assessed using profitability, decision stability, fairness-oriented pricing behavior, decision consistency, and interpretability. To improve transparency, trajectory-based policy audits and SHapley Additive exPlanations were applied to identify the main factors influencing pricing decisions. The results show that the Deep Q-Network agent achieved the most balanced performance, increasing total profit by 12.58% while recording no unethical price increases under low-demand conditions. Explainability analysis showed that stock or resource levels, demand shifts, and price elasticity were the strongest positive drivers of pricing actions, whereas inventory hoarding and unfavorable price increases reduced decision quality. The findings indicate that reinforcement learning can support sustainable and resilient urban decision-making when optimization objectives are combined with trustworthy AI principles. The proposed framework provides a practical basis for accountable AI-based decision-support systems in smart city operations, including demand-responsive services, resource optimization, sustainable dynamic pricing, and energy-aware management.

Cpu scheduling from classical heuristics to learning-based and multicore paradigms: a systematic review, comparative analysis, and research roadmap

Comparative Finite Element Modal Analysis of Reinforced Concrete Cantilever Beams with Four- and Six-Bar Longitudinal Reinforcement Layouts

This study compares the elastic modal characteristics of reinforced-concrete cantilever beams containing four and six 12 mm longitudinal reinforcing bars using three-dimensional finite element analysis (FEA) and an exploratory machine learning (ML) exercise. Beam geometry, material properties, bond assumptions, and boundary conditions were held constant. Mesh refinement from a 50 mm to a 25 mm nominal element size changed the fundamental natural frequency by 0.18%, satisfying the adopted 1% convergence criterion. The Fine mesh FEA fundamental frequency of 36.048 Hz differed by 1.49% from the Euler–Bernoulli transformed-section estimate of 35.52 Hz. Block Lanczos extraction provided the first six natural frequencies and corresponding normalised mode shapes. Increasing the longitudinal steel area from 452.39 to 678.58 mm2 changed the calculated frequencies by −0.02% to +0.19%, while the two layouts exhibited similar mode-shape topology under the assumed intact, linear-elastic, perfectly bonded conditions. Four regression algorithms were fitted to the 12 correlated mode–configuration records using only mode number and reinforcement count as inputs and natural frequency as the output. The reported R2, MAE, and RMSE values describe complete dataset fitting or interpolation and do not establish generalisation to unseen configurations. No physical modal displacement, strain, or stress amplitude is reported because eigenvector scaling is arbitrary and no forced-response analysis with defined excitation and damping was performed.

Cholinergic-dependent dopamine signals in mouse dorsomedial striatum are regulated by frontal but not sensory cortices

Abstract Everyday decisions depend on associations between sensory stimuli, actions, and outcomes. The striatum supports these sensorimotor associations through dopamine-dependent plasticity. Recent work has characterized a local striatal microcircuit in which cholinergic interneurons (CINs) modulate dopamine release via activation of nicotinic receptors on dopamine axons. Here, we show that visual stimuli evoke dopamine in the dorsomedial striatum partly through this cholinergic mechanism. Using anatomical and functional methods to identify the pathways involved, we found that visual and auditory cortices lack connectivity with CINs and were unable to drive cholinergic-dependent dopamine release. Frontal regions, which were activated by visual stimuli, strongly recruited CINs, producing robust dopamine release both ex vivo and in vivo. These findings reveal a fundamental distinction between sensory and frontal corticostriatal inputs, demonstrating that only the latter can evoke cholinergic-dependent dopamine signals. This work establishes a framework for understanding how cortical circuits shape striatal dopamine to support reinforcement learning.

Correction to “Research on autonomous decision‐making method for spacecraft in the mission of rendezvous and approaching to maneuvering target based on deep reinforcement learning”

C. Huang, A. Xing, Q. Zeng, and F. Xiong, Research on autonomous decision-making method for spacecraft in the mission of rendezvous and approaching to maneuvering target based on deep reinforcement learning, Asian J. Control 27 (2025), 2541–2557, DOI https://doi.org/10.1002/asjc.3599. In the Funding section, the funding source “the Natural Science Foundation of Heilongjiang Province” was missing. The updated Funding section is as follows: Funding information: “This work was supported by the Natural Science Foundation of Heilongjiang Province (Grant No. LH2023F032) and the National Natural Science Foundation of China – Young Scientist Fund (Grant No. 52102455).” We apologize for this error.

Dynamic Hippocampal–Striatal Information Flow Accompanies Behavioral Strategy Transitions During Sequential Learning in Pigeons: A Preliminary Study

Sequential decision-making requires animals to flexibly balance model-based (MB) and model-free (MF) strategies to adapt to changing environments. The hippocampus (Hp) and striatum (ST) are two important components of the broader neural networks supporting these processes; however, how their dynamic interactions reorganize during learning-dependent strategy transitions remains poorly understood. Here, we trained pigeons on a two-step sequential decision-making task while simultaneously recording local field potentials (LFPs) from the Hp and ST. A dynamic reinforcement learning framework combined with a sliding-window approach was used to characterize temporal changes in behavioral strategies, and phase transfer entropy (PTE) was applied to estimate directed information flow between the Hp and ST across theta, beta, and broad gamma (30–80 Hz) frequency bands. Behavioral modeling revealed a gradual transition from early MB-like, task-structure-sensitive control toward later MF-like value-guided behavior as learning progressed. PTE analysis demonstrated a consistent Hp-to-ST directional bias across all analyzed frequency bands during task acquisition. Notably, gamma-band Hp-to-ST information flow exhibited a consistent decline over training, whereas theta- and beta-band interactions showed less consistent changes across individuals. Additional analyses showed that relative MB model evidence and gamma-band Hp-to-ST information flow covaried across learning, but this association was no longer significant after controlling for learning progression, indicating parallel rather than independently coupled changes. These preliminary findings indicate that hippocampal–striatal communication undergoes frequency-specific reorganization during sequential learning. The reduction in gamma-band Hp-to-ST information flow accompanies, rather than independently predicts, the behavioral strategy transition, suggesting learning-related modulation of interregional coordination as task demands change.

IC-VTP: An intent-conditioned reinforcement learning method for robust long-term trajectory prediction of non-cooperative vessels

Harnessing Artificial Intelligence for Strategic Retail Promotions

The retail industry is undergoing a transformation due to the integration of Artificial Intelligence (AI) in promotional strategies. Traditional methods like print ads, radio commercials, and in-store displays face challenges in today's dynamic retail environment, such as the inability to integrate with online platforms and provide interactive experiences. AI technologies like machine learning, natural language processing, computer vision, and reinforcement learning are redefining promotional strategies, enhancing customer engagement, and optimizing marketing efforts. AI-driven promotions are set to revolutionize the industry, with hyper-personalization and omnichannel retailing becoming key trends. However, challenges such as data privacy concerns, algorithmic biases, and the need to balance automation with human touch points are also addressed. The chapter provides strategic recommendations for retailers to harness AI effectively, including investing in cutting-edge tools and fostering collaborations with technology providers.

Comparison of Reinforcement Learning-based Algorithm for Eco-Driving Control in Electric Vehicles

A Reinforcement-Learning-Based Fuzzy PID Control Strategy for the Unwinding Tension System of a Lithium-Ion Battery Coating Machine

Stable tension is critical for the coating quality of lithium-ion battery electrodes. As the origin of tension control, the unwinding system’s control accuracy governs the stability of downstream processes and the final yield. To achieve the required precision, we propose a control strategy that combines reinforcement learning and fuzzy PID. We first derived a nonlinear time-varying dynamic model of the unwinding tension system based on the unwinding mechanism. Leveraging this model, we then designed a reinforcement-learning-based fuzzy PID controller. Finally, we validated the performance of the proposed control strategy through simulations and experiments. Simulations and experiments confirm that, under varying coil radius, the reinforcement-learning-based fuzzy PID controller outperforms both the conventional PID and fuzzy PID controllers in robustness, effectively accommodating the effects of time-varying tension system parameters. Moreover, this method substantially improves the dynamic performance of the unwinding system, with pronounced overshoot suppression, and demonstrates superior robustness and disturbance rejection capabilities.

Deep reinforcement learning based trajectory optimization in unmanned aerial vehicles: a lyapunov control perspective