Dual-Stream Self-Supervised Synergistic Network for Robust RGB-T Fusion and Tracking

Abstract

RGB-T tracking is a challenging task due to the various obstacles, including occlusion, illumination, thermal crossover, and so on. However, existing research focuses purely on multimodal interaction or fusion design, while overlooking the effective handling of robustness in the face of challenges. Additionally, these paradigms naively fuse the two modals, without any contemplation of a collaboratively dominated fusion scheme. To address these issues, we propose DS4Net, a Dual-Stream Self-Supervised Synergistic Network for robust RGB-T fusion and tracking. First, unlike the traditional One-Stream paradigm, we introduce a novel Dual-Stream architecture, with two distinct versions of RGB-T modalities as input for self-supervised training to achieve balanced target representation learning and localization, thus addresses target appearance deviations caused by challenging scenarios. Then, to effectively bridge two streams, we design a Modality Reciprocal Adapter (MRA), which enhances the synergistic interaction of target-appearance cues and target-background information between streams during the training phase. Furthermore, to enhance robust fusion for tracking, we introduce Collaboratively Dominated Multimodal Fusion (CDMF), aiming at learning the fluctuating identities of modalities under challenging conditions provided by the Dual-Stream and collaboratively leveraging the dominant and auxiliary modalities to realize a more robust fusion. Finally, extensive experiments conducted on three RGB-T benchmark datasets, i.e., LasHeR, RGBT234, and RGBT210, demonstrate the excellence and superiority of our proposed method. In particular, we achieve 71.1% PR on LasHeR, 88.4% PR on RGBT234 and 87.1% on RGBT210, outperforming the popular approaches.