Render-to-Optimize: Scene Coordinate Regression with Differentiable Gaussian Splatting for Camera Relocalization
Abstract
3D Gaussian Splatting (3DGS) demonstrates exceptional potential in localization due to its realistic rendering. However, existing methods typically confine it to pose refinement, making their performance highly dependent on the quality of the initial pose estimate. This dependency limits their robustness when the initialization is inaccurate. In this paper, we introduce a new Render-to-Optimize relocalization paradigm, shifting the focus from optimizing camera poses to directly optimizing the scene coordinate regression network. Built on this idea, we present DiffGS-SCR, a differentiable Gaussian-based pipeline that directly optimizes the Scene Coordinate Regression (SCR) network for improving camera relocalization. The DiffGS-SCR pipeline proceeds in two stages. In the first stage, we propose a Render-Aware Geometric Optimization (RAGO) scheme to refine the coordinate regression network by leveraging differentiable rendering for photometric supervision. To ensure a robust geometric initialization, we further introduce a Geometry-Guided Sampling mechanism that filters out low-quality Gaussian primitives based on their predicted attributes, enabling more accurate pose estimation. In the second stage, we perform pose refinement using Feature Gaussian. By synthesizing dense feature maps and establishing dense feature matches with the query image, effectively optimize the initial pose. Extensive experiments demonstrate that DiffGS-SCR achieves state-of-the-art performance, significantly outperforming existing SCR and 3DGS methods in both accuracy and robustness. The code is available at https://github.com/86057757comeon/DiffGS-SCR.