Spatio-Temporal Masked Autoencoders with Contrastive Cross-Attention for Robust Video Semantic Segmentation in Adverse Autonomous Driving Conditions

Abstract

Autonomous vehicle perception systems rely heavily on robust semantic segmentation to interpret complex urban environments under dynamic driving conditions. While modern vision transformers (ViTs) demonstrate remarkable performance on pristine benchmarks, their accuracy degrades drastically in adverse weather conditions (e.g., dense fog, heavy precipitation, nighttime glare, and optical motion blur) and sparse temporal settings. In this paper, we propose ST-MaskContrast, a novel self-supervised spatio-temporal masked autoencoder framework tailored for robust real-time video semantic segmentation. ST-MaskContrast incorporates a decoupled space-time token masking strategy (masking up to 80% of video frame patches) coupled with a multi-scale cross-attention temporal decoder that enforces invariant representations between clean and corrupted sequential frames via contrastive patch-level loss. We evaluate ST-MaskContrast across three challenging public driving benchmarks: Cityscapes, KITTI-360, and the adverse-weather ACDC (Adverse Conditions Dataset with Correspondences). On the ACDC dataset, ST-MaskContrast achieves a state-of-the-art mean Intersection over Union (mIoU) of 78.4%, outperforming current supervised vision transformers (SegFormer, Mask2Former) by +6.2% mIoU while reducing inference latency by 34.8% through sparse token pruning. Extensive ablation studies confirm that our joint contrastive reconstruction objective dramatically mitigates domain shift, providing unprecedented perceptual resilience for safety-critical autonomous navigation.