Illumination-Robust Stereo Semantic Occupancy for Autonomous Driving via Cross-Modal Fusion

Abstract

Three-dimensional semantic scene completion (SSC) is fundamental for autonomous driving perception systems, providing dense voxel-wise representations of the environment that capture both occupied and free space along with semantic labels. However, existing vision-based methods exhibit significant performance degradation under challenging illumination conditions such as low-light environments, strong shadows, and severe backlighting. This paper proposes a stereo-based SSC framework that achieves illumination-robust 3D occupancy prediction by exploiting color and grayscale stereo imagery. A vision foundation model backbone is shared by the two modalities, and two cross-modal fusion mechanisms are introduced to keep both the appearance and the geometry reliable when the illumination degrades. The first mechanism is Reliability-weighted Cross-modal Fusion, in which the grayscale stream serves as a second value stream within a three-dimensional deformable cross-attention. Grayscale evidence is therefore added where the color appearance is unreliable. The second mechanism is Grayscale Stereo-depth Fusion, which fuses a grayscale stereo-depth volume with the color stereo-depth volume. Scene geometry is therefore preserved when color stereo matching fails under adverse illumination. Extensive experiments on the SemanticKITTI dataset show that the proposed method achieves 44.96% Scene Completion Intersection over Union and 16.98% mean Intersection over Union (mIoU) on the hidden test server. Under a controlled illumination degradation protocol, the proposed method retains 42% of its clean-input mIoU at the most severe level, whereas existing color-only methods retain 18% to 22%. Against a color-only control trained with the identical degradation augmentation, the robustness is attributed to the fusion rather than to the training procedure. The code and the trained model are available at https://github.com/YS-Jiang/illumination-robust-ssc Note to Practitioners—This work addresses a critical safety challenge for camera-based autonomous driving, where existing perception methods lose 78% to 82% of their accuracy under adverse lighting conditions such as shadows, low-light environments, and backlighting. The proposed framework maintains 42% of its normal-lighting accuracy at the most severe degradation level tested, by leveraging color and grayscale cameras already available in many platforms, requiring no additional hardware. Grayscale imagery keeps a higher signal-to-noise ratio in low light, and two fusion mechanisms combine color and grayscale evidence for both appearance and geometry. A key practical benefit is the modular design of these mechanisms, making them applicable as additions to other stereo vision-based perception pipelines. The benefit comes from the second physical sensor, so converting a color image to grayscale is not a substitute, and a platform without a grayscale pair cannot use the method directly. The proposed approach provides a cost-effective solution for safe, resilient and adaptable autonomous driving in complex illumination environments.