This work extends the NTK framework to reinforcement learning (RL) by introducing a temporal-difference flow formulation that accommodates time-varying mini-batches and TD targets, yielding a closed-form solution for value function evolution in an overparametrized setting.
Abstract
Despite their widespread success across machine learning, the learning dynamics and generalization properties of neural networks remain poorly understood. The neural tangent kernel (NTK) provides an analytical characterization of the learning dynamics of infinitely wide neural networks and serves as a useful proxy for finite-width networks in supervised learning. In this work, we extend the NTK framework to reinforcement learning (RL) by introducing a temporal-difference (TD) flow formulation that accommodates time-varying mini-batches and TD targets, yielding a closed-form solution for value function evolution in an overparametrized setting. We also devise an optimization-based state-space exploration method that leverages the closed-form solution, coupled with constraints on the environment, to model the set of feasible states. On a deterministic grid environment, we validate that the proposed RL-based NTK formulation is indeed learning efficiently and adapting to the distribution shift in the training data distribution, and we demonstrate that the resulting exploration strategy improves sample efficiency. To our knowledge, this is the first work to extend NTK theory to characterize the learning dynamics of RL training with time-varying mini-batches and TD updates.
Deep reinforcement learning (DRL) relies on neural networks trained under temporally correlated experience, evolving replay distributions, and bootstrapped targets, conditions that can destabilize neural function approximation. This study investigates controllable orthogonalization as a network-level mechanism for impr...
Jia Yu, Dun Su, Hua Cui et al.· Informatics· 0 citations
This work investigates conditions that ensure fast transfer in the growing-horizon regime of hyperparameter transfer in a shallow linear network with a single trainable hidden matrix, trained by full-batch gradient descent.
Mana Sakai, Masaaki Imaizumi· arXiv.org· 0 citations
Under the unified modeling framework, the relaxations that prior art made during building the advantage-embedded regression target are unraveled and approximating the KKT condition and posterior normalizer for the linear and exponential tilt shapes DiffusionNFT and FlowAWR respectively, while preserving the exact spars...
Toyota Li, David Zhao, Alan Zhao· arXiv.org· 0 citations
Two tractable ways of modeling distributional returns in MORL are proposed and combined with recent advances in function approximators for reinforcement learning show improvements in function approximation that substantially improve the quality of the produced solution sets.
The paper investigates how the choice of a generative stochastic model of the training environment affects the numerical stability of the learning and the transfer of the Double DQN policy to real time series. Four environments — GBM, GARCH(1,1)-t, Minimal Neural SDE, and Neural Jump-Diffusion — are compared, created o...
Y. Yurchenko, Oleksandr Zakovorotnyi· Journal of Mathematical Mode...· 0 citations
Stochastic gradient descent (SGD) is the cornerstone of modern deep learning, yet its theoretical convergence properties remain poorly understood in non-convex settings. To address this gap, we study SGD dynamics in deep linear neural networks - simplified models that retain key non-convex characteristics while remaini...
Gabin Maxime Nguegnang, Bubacarr Bah, Holger Rauhut et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.