This work introduces a constrained heuristic-learning protocol in which an agent iteratively proposes, evaluates, and revises controller implementations while interacting exclusively through the public benchmark interface, and suggests that heuristic learning through coding agents constitutes a credible and complementary alternative to conventional reinforcement learning.
Abstract
Active flow control involves nonlinear dynamics, partial observations, and computationally expensive simulations, making controller design particularly challenging. Deep reinforcement learning (DRL) has emerged as a powerful framework for such problems, but its success typically relies on large numbers of simulator interactions and produces neural-network policies whose decision process often remains difficult to interpret. In this work, we investigate a different paradigm: instead of optimizing neural-network parameters, we use modern coding agents to search directly for explicit executable feedback laws. We introduce a constrained heuristic-learning protocol in which an agent iteratively proposes, evaluates, and revises controller implementations while interacting exclusively through the public benchmark interface. The proposed framework is evaluated on 13 active flow-control benchmarks spanning one, two, and three-dimensional problems and compared against the strongest available DRL baselines under identical simulation budgets. The discovered heuristic controllers match or outperform the best DRL policy in 10 of the 13 environments while remaining compact, interpretable, and directly inspectable. Beyond aggregate performance, the resulting controllers reveal physically meaningful feedback mechanisms, transfer successfully across more challenging configurations, and remain competitive under varying Reynolds and Rayleigh numbers, actuator counts, and observation sparsity. These results suggest that heuristic learning through coding agents constitutes a credible and complementary alternative to conventional reinforcement learning, combining competitive performance with physically interpretable controller representations. Prompts and source code are available at https://github.com/DonsetPG/fluid-heuristic-learning.
Operating constrained dynamical systems requires controllers to efficiently solve complex tasks while enforcing recursive feasibility and safety constraints. To address these competing requirements, we present Feasible Action for Optimal Control (FAOC), a novel control framework integrating Reinforcement Learning (RL) and Optimal Control (OC). The key contribution is a computationally efficient, optimization-based mapping algorithm that transforms the RL agent's action from a static abstract set into a state-dependent feasible parameter set of the Optimal Control Problem (OCP), guaranteeing strict satisfaction of the dynamical system's constraints. Thus, FAOC effectively combines the predictable safety of OC with the flexibility of RL. In contrast to prior work, the abstract action space of the RL agent does not require expert or heuristic design, and the OCP formulation is not compromised by the inability of RL to guarantee feasibility. We apply our approach to real-time motion planning for robot table tennis, which encapsulates these challenges. Via simulated experiments, we show that FAOC outperforms state-of-the-art baselines in both sample efficiency and closed-loop performance.
Stefan Richter, Alberto Giammarino, G. Torrente et al.· 0 citations
PEARL employs an actor-adjoint algorithm that leverages automatic differentiation to compute policy gradients over short horizons and adjoint-based sensitivities of future returns approximated via neural networks, significantly reducing the number of environment interactions, while mitigating long-term gradient instabilities.
Matteo Tomasetto, Nicolò Botteghi, Gabriele Bruni et al.· 0 citations
Controller synthesis is a promising approach as a planner for self-adaptive systems, as it can automatically re-synthesize control strategies that satisfy the specified properties in response to runtime changes. To enhance efficiency, Directed Controller Synthesis prunes the search space by incrementally constructing a partial view of the system, aiming to find a valid controller without exhaustive exploration. This process is steered by an exploration policy (i.e., heuristic), and Reinforcement Learning has proven highly effective for learning such policies. However, a key challenge is anisotropic generalization, i.e., a policy trained on specific domain parameters is specialized, performing well in certain scenarios while remaining fragile in others. To this end, we propose a Mixture-of-Experts framework that combines multiple policies, leveraging their complementary strengths to form a more robust exploration policy. The evaluation on the Air Traffic benchmark shows that our proposal significantly increases the number of solvable instances.
Toshihide Ubukata, Mingyue Zhang, Zhiyao Wang et al.· SEAMS@ICSE· 1 citation
Transition state (TS) search is a crucial step in understanding chemical reactivity and mechanisms, yet conventional algorithms remain computationally intensive and heavily reliant on initial guesses, user s expertise, and chemical intuition. While recent machine learning approaches have shown promise, they demand either large training datasets or geometric interpolation between known endpoints, limiting their generality. In this work, we introduce a TS search model based on the soft actor-critic model, an advanced reinforcement learning algorithm in which an agent learns to navigate potential energy surfaces directly from local energetic and curvature information starting from a given reactant and its corresponding product. By formulating the search as a sequential decision-making process in internal coordinates, the agent adaptively proposes chemically meaningful structural updates through a reward function designed to promote movement towards saddle point regions. Without labelled trajectories or prescribed reaction pathways, the method successfully identifies TS geometries for standard benchmark reactions, operating directly on realistic molecular potential energy surfaces. These results highlight the potential of RL as a general strategy for reducing dependence on initial guesses and enabling scalable, automated reaction discovery across diverse chemical systems.
Utham Suresh, Konstantinos D. Vogiatzis· 0 citations
Optimal decision-making under uncertainty is a shared challenge across modern chemical, manufacturing, and energy systems that increasingly demand safe, data-driven autonomy. This talk revisits optimal control through the lens of the Bellman equation, emphasizing how optimal control theory and reinforcement learning have developed complementary, yet largely disconnected, perspectives on global optimality. In one view, central to reinforcement learning, the Bellman equation defines a global optimality condition that guides iterative policy learning from interacting with the system, but typically yields opaque control laws that are difficult to interpret, and deploy in safety-critical settings. In another view, widely adopted in model predictive control (MPC), the Bellman equation underpins tractable finite-horizon optimizations that deliver interpretable, constraint-aware, and modular local controllers, yet without explicit guarantees on alignment with global optimality. Building on these ideas, we introduce a local–global paradigm that treats MPC and related optimization-based controllers as structured function approximators designed to approximately satisfy the global Bellman optimality condition. We discuss algorithmic strategies for learning interpretable local decision makers whose adaptation is guided by Bellman residuals, along with the benefits and practical challenges that arise in terms of stability, constraint satisfaction, and sample efficiency. These concepts are illustrated through case studies that unify reinforcement learning and MPC for safe, high-performance control in complex, uncertain dynamical systems. The talk concludes by outlining open problems and research opportunities in learning interpretable control policies that achieve globally optimal performance while retaining the transparency and reliability required for real-world process control and optimization applications.
A. Mesbah· Proceedings of the 3rd Found...· 0 citations
Actor-critic models are a class of model-free deep
reinforcement learning (RL) algorithms that have
demonstrated effectiveness across various robot learning
tasks. While considerable research has focused on improving
training stability and data sampling efficiency, most
deployment strategies have remained relatively simplistic,
typically relying on direct actor policy rollouts. In
contrast, we propose PACHS (Parallel Actor-Critic Heuristic
Search), an efficient parallel best-first search algorithm
for inference that leverages both components of the
actor-critic architecture: the actor network generates
actions, while the critic network provides cost-to-go
estimates to guide the search. Two levels of parallelism
are employed within the search---actions and cost-to-go
estimates are generated in batches by the actor and critic
networks respectively, and graph expansion is distributed
across multiple threads. We demonstrate the effectiveness
of our approach in robotic manipulation tasks, including
collision-free motion planning and contact-rich
interactions such as non-prehensile pushing. Visit
https://p-achs.github.io for demonstrations and examples.
Itamar Mishani, Hanlan Yang, Luca Pivetti et al.· Proceedings of the Internati...· 0 citations