Reconstruction-Free Vision-Based Control via Differentiable Long-Horizon Influence

Abstract

Vision-based reinforcement learning has emerged as a promising approach for autonomous robotic control. It enables closed-loop policy learning directly from high-dimensional visual observations. However, existing methods often learn visual representations through prediction or reconstruction objectives, which may preserve visual details with limited relevance to control. To address this issue, we propose a reconstruction-free vision-based control framework based on differentiable long-horizon influence. The key idea is to evaluate how future control would change when different parts of the visual representation are removed. We compare a nominal latent rollout with counterfactual rollouts in which individual latent components are suppressed. The resulting changes in predicted return, safety cost, and trajectory evolution define long-horizon influence scores. These scores are further converted into differentiable gate supervision, allowing future closed-loop effects to guide visual abstraction and encouraging the retention of control-critical information. Experiments in CARLA and Adroit demonstrate consistent improvements across autonomous driving and dexterous manipulation tasks. Real-world deployment on a physical Clearpath Jackal robot further demonstrates safe and real-time closed-loop control. Note to Practitioners—Autonomous robots that rely on cameras often operate in visually cluttered environments, where not all image details are useful for control. In practice, preserving unnecessary visual information may increase model complexity and may distract the controller from safety-critical cues such as obstacles, traversable regions, lanes, or moving agents. This paper proposes a training framework that helps the robot learn which latent visual components should be retained for closed-loop control. During training, the method estimates whether removing a latent component would change future control performance or safety-related behavior, and uses these estimates to train a compact visual abstraction module. During deployment, counterfactual rollouts are not executed; the robot only runs the encoder, abstraction gate, and policy network as a high-level visual controller. Practitioners can apply the method when representative visual trajectories and safety-cost signals are available. Practical use still requires careful data collection, safety-cost design, and validation under the target operating conditions.