Depth Camera or RGB-to-Depth? A Study of RGB-to-Depth Estimation for Robot Policy Learning

Abstract

Point cloud–based robot policy learning has emerged as a powerful approach for robotic manipulation by enabling policies to directly exploit 3D geometric structure. In such pipelines, depth image acquisition plays a critical role in point cloud construction. However, raw depth from depth cameras is often noisy, sensitive to transparent, reflective, and repetitive surfaces, and limited by a minimum sensing distance. Recent advances in learned depth estimation offer a promising alternative: modern monocular, stereo, and multi-view approaches have achieved impressive performance in vision benchmarks. However, robotics still lacks a systematic understanding of how these advances translate to end-to-end policy learning and execution. In this work, based on our introduced Depth-Estimated Imitation Learning (DEIL) general framework, we present the first systematic study of how modern depth estimation impacts point cloud–based imitation learning across two paradigms: In-Context Imitation Learning (ICIL) and Behaviour Cloning (BC). Within each paradigm, we keep the point cloud reconstruction procedure, policy backbone, and evaluation protocol fixed across depth sources. Through 71,800 simulated and 160 real-world rollouts, several depth estimators achieve reasonable downstream performance in simulation, although a consistent gap to GT depth remains. In real-world ICIL, the best-performing multi-view estimator is competitive with the tested RGB-D cameras on normal tasks and achieves higher success when depth sensing is unreliable. Our failure mode analysis further shows that reconstructed geometric quality generally correlates with policy success, with flying points being particularly harmful to BC. These findings highlight the need for improved geometric awareness and stability before learned depth estimation can be broadly deployed in robot manipulation.