HSMTrack: Heterogeneous-State Motion Tracking for Vision-Sensor Pipelines

Abstract

Motion-only multiobject tracking (MOT) suffers from ID switches in uniform-appearance and deformation-heavy scenes. In these settings, appearance cues become less reliable, so stable identities depend mainly on motion information. Existing methods often process all bounding-box variables together, which can weaken cues needed for prediction and matching. We address this problem by treating each trajectory as a heterogeneous multivariate time series (MTS) and redesigning the motion-only pipeline for embedding, encoding, and matching. HSMTrack separates box-state variables before modeling their temporal and cross-variable relationships, then uses deformation-aware matching for identity association. The method requires no appearance branch and can serve as a post-detection motion module in vision-sensor tracking pipelines. Its SSM-based encoder has linear complexity with respect to trajectory length, reducing modeling cost compared with attention-based alternatives. HSMTrack achieves 59.6 IDF1 and 42.9 AssA on DanceTrack, and 77.9 IDF1 and 67.2 AssA on SportsMOT. Under a unified end-to-end protocol, it reaches 34.1 frames/s on RTX 4090 and 10.5 frames/s on Jetson Orin NX.