Skip to content
Preprint

A Model-Free Continuous-Time RL Algorithm from Discrete-Time Off-Policy Data with End-to-End Guarantees

Oct 2026 · 0 citations · 30 references
Computer Science Mathematics

TL;DR

This paper develops a model-free PhiBE policy iteration framework that uses a single state-action critic while preserving the continuous-time action-ranking scale and identifies a bias-variance trade-off: smaller sampling intervals improve the approximation to the continuous-time problem but can increase statistical error at a fixed sample size.

Abstract

In this paper, we study continuous-time reinforcement learning, where the state follows an unknown stochastic differential equation but only a fixed off-policy batch of discrete-time transitions is available. Standard discrete-time Q-learning becomes problematic as the sampling interval shrinks, because differences between action values vanish. Continuous-time alternatives, such as Deep Advantage Updating and martingale q-learning, preserve the action signal but jointly learn a value function and an action-dependent critic. Building on the PhiBE-Q formulation (arXiv:2606.21925), we develop a model-free PhiBE policy iteration (PhiBE-PI) framework that uses a single state-action critic while preserving the continuous-time action-ranking scale. At each iteration, the framework learns the critic directly from sampled increments and uses it to improve the policy. The same batch is reused throughout, without explicitly estimating the drift or diffusion. We further develop an implementation under linear function approximation and establish a local end-to-end suboptimality guarantee for the learned policy. Our analysis quantifies how errors from time discretization, function approximation, and finite-sample policy evaluation propagate through policy iteration. It accounts for the dependence of successive policies on the reused data, so the guarantee does not require a fresh batch at each iteration. Moreover, we identify a bias-variance trade-off: smaller sampling intervals improve the approximation to the continuous-time problem but can increase statistical error at a fixed sample size. Thus, finer time discretization does not necessarily lead to better policy performance.

View source

Similar papers

Preprint Oct 2026

Continuous-Time Policy Gradient from Discrete Data: Convergence Analysis and Model-Free Algorithms

This work develops a policy gradient framework that combines rigorous convergence analysis with learning from discrete data and integrates this framework with PhiBE, which constructs a continuous-time surrogate generator from discrete transition data.

Martín Hernández, Xin-Yan Qing, Mo Zhou et al. · 0 citations
#artificial intelligence Preprint Nov 2026

Horizon-Scalable Q-Learning

Learning accurate off-policy state-action value functions (Q-functions) from offline datasets becomes harder as the task horizon grows. One-step temporal-difference (TD) learning bootstraps after every transition, so small per-step errors accumulate along the horizon, and early states can be badly misvalued even when t...

Hao-Ran Xu, Pragya Srivastava, Xin-Jie Liu et al. · 0 citations
Preprint Sep 2026

Model-free Reinforcement Learning for Continuous Time and State: A Stochastic Maximum Principle Approach

This paper develops a model-free reinforcement learning (RL) algorithm based on the stochastic maximum principle for continuous-time stochastic control problems with continuous state and action spaces. For a parameterized Markovian policy, we establish the existence of the decoupling field for the adjoint backward stoc...

Li-Jun Bo, Yi-Jie Huang, Jing-Fei Wang · 0 citations
#machine learning Preprint Oct 2026

Drift, Then Spread: Learning One-Step Generative Policies for Online Reinforcement Learning

Generative policies offer expressive action distributions for online reinforcement learning (RL), but many diffusion- and flow-based policies require costly iterative sampling. We propose DriftSpread, a one-step generative policy based on Drifting for online maximum-entropy (MaxEnt) RL. To incorporate entropy without a...

Chun-Yu Luo, Hong-Yu Zhao, Guang-Yi Ni et al. · 1 citation
#machine learning Preprint Oct 2026

Reinforcement Learning with Multi-Step Transition Predictions: A Bellman Theory

Multi-step predictions of future transition realizations can improve reinforcement learning, but their value remains difficult to quantify. Directly including these predictions in the state can lead to exponential state-space growth, making direct dynamic programming computationally expensive. We study discounted finit...

Chen-Bei Lu, Zai-Wei Chen, Tong-Xin Li et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.