#artificial intelligence #machine learning Preprint

On the Convergence of Semi-Gradient TD: A Mean Field Perspective

Haruki Settai Tadashi Kozuno Shinji Ito

Oct 2026 · 0 citations · 15 references

Computer Science

TL;DR

Under this condition, it is proved exponential convergence to a unique fixed point and its bias relative to the unregularized Bellman solution and it is shown that the same mechanism extends to entropy-regularized off-policy control through soft Q-learning.

Abstract

Semi-gradient temporal-difference learning is a core algorithmic component of reinforcement learning, but bootstrapping with function approximation generally induces a non-conservative vector field and can lead to instability. We analyze this effect for two-layer neural networks in the mean-field regime and identify a simple stability mechanism in which entropy-induced relaxation dominates the sensitivity of the moving bootstrap target. Under this condition, we prove exponential convergence to a unique fixed point and quantify its bias relative to the unregularized Bellman solution. We also show that the same mechanism extends to entropy-regularized off-policy control through soft Q-learning.

DOI arXiv Semantic Scholar

View source

Similar papers

#machine learning Preprint Sep 2026

Exact information accounting for SGD methods

An exact, information-theoretic analysis of stochastic gradient descent and its variants is given, showing that a preconditioned SGD step is the posterior-mean update of a Gaussian Bayes model, and that its one-step regret splits into an intrinsic-time cost and a change in comparator information.

Akshay Balsubramani · 0 citations

Finite-Time Analysis of Minimax Q-Learning

Narim Jeong, Donghwan Lee · 0 citations

#machine learning Preprint Oct 2026

Mind the Drift: Diagonal Linear Networks Under Large Learning Rates

This analysis reveals that large learning rates do not universally hinder the recovery of sparse solutions and can be harnessed to control the implicit bias of training.

Aniket Sanyal, Tom Jacobs, Rebekka Burkholz · 0 citations

HANQ: Hypergradients, Asymmetry, and Normalization for Fast and Stable Deep Q -Learning

Braham Snyder, Chen-Yu Wei · 0 citations

Viscosity Convergence Analysis for Deep Q-Networks

This work studies a spatially-coupled monotone ResNet architecture whose one-step operator is a non-negative local aggregation, designed to satisfy monotonicity and stability at the operator level, and obtains operator-level convergence to the viscosity solution via the Barles–Souganidis framework.

Qian Qi · 1 citation

#machine learning Preprint Oct 2026

Entropy-Regularized Mean-Field Games with Average-Reward Criterion: Span-Seminorm Contractions and Model-Free Learning

We study stationary discrete-time mean-field games with finite state and action spaces under the average-reward criterion, with an entropy-regularized objective and population-dependent dynamics and rewards. Under a minorization condition, the soft Q-operator is a span-seminorm contraction for the full transition kerne...

Gunes Akbas, Naci Saldi, Berkay Anahtarci et al. · 0 citations