Robust Person Following in Crowds Using Complementary Cross-Modal Fusion

Abstract

Person following in crowded indoor environments remains challenging owing to occlusion-induced tracking loss and identity confusion among bystanders. We propose a crossmodal person-following system that couples three components: First, a video object segmentation tracker, initialized by a single click on the target person, propagates identity-stable masks without requiring repeated per-frame detection. Second, a parallel 2D LiDAR leg tracker projects its position estimate onto the image plane to support recovery when visual tracking becomes unreliable. Third, a dual-model Kalman filter with depth-scaled measurement noise maintains an accurate three-dimensional state estimate, adapting between steady walking and sudden maneuvers. In experiments across various occlusion conditions, the proposed system achieved up to a 31-point improvement in tracking availability and up to a 72-point reduction in non-tracking time ratio compared to conventional camera-based tracking. The proposed system yields a highly robust tracking pipeline that effectively reduces the chance of permanent target loss, even during prolonged visual interruptions.