Robust 3D-Aware Video Object Tracking for Mobile Robots Based on 2D Vision Foundation Models

Abstract

Egocentric cameras are widely used in robotic navigation and manipulation, yet conventional 2D Video Object Tracking (VOT) methods suffer from severe performance degradation under rapid viewpoint changes and frequent frame-out events. Because most existing trackers rely solely on 2D appearance cues, they often fail to recover object identities once targets temporarily disappear. We propose R3DVOT, a 3D-aware tracking framework that augments 2D vision foundation models with spatial geometric reasoning. Its core component, the Position-Aware Memory Selection (PAMS), lifts mask candidates into a canonical 3D world coordinate system and maintains a persistent world-frame state for position-consistent hypothesis selection. By curating the memory bank with spatially consistent anchors, R3DVOT improves robustness to occlusion and frame-out events. On the VOT benchmark, R3DVOT achieves an AUC improvement of 5.1% over SAM 2 and 1.4% over SAMURAI. Furthermore, on the VOS benchmark, R3DVOT increases the J&F score by 4.6% compared to SAM 2 and 9.0% compared to SAMURAI. These results highlight the effectiveness of 3D spatial continuity in enhancing tracking, segmentation, and long-term identity consistency in robotic perception.