Frequency-Guided Cross-Modal Distillation With Illumination-Invariant Prototype Alignment for RGB-Infrared Object Detection
Abstract
Multi-modal object detection using red-green-blue (RGB) and infrared (IR) imagery is essential for safety-critical applications under diverse illumination conditions, such as autonomous driving and urban surveillance. Recent RGB-IR detectors have improved detection performance through feature-level fusion strategies, including attention-based fusion, illumination-aware fusion, and cross-modal alignment. Mono-Modality Distillation (M2D) mitigates mono-modality insufficient learning by transferring mono-modal teacher knowledge to multi-modal backbones. Local Illumination-aware Fusion (LIF) complements this by adaptively weighting RGB and IR features according to local illumination conditions. Together, they achieve strong detection performance over existing methods. However, two limitations remain. First, the fused representation is mainly optimized by the detection loss and is not explicitly guided to reflect which modality provides more reliable information in each frequency band. Second, illumination-aware fusion may produce illumination-dependent feature distributions, where same-class objects form different sub-clusters according to local lighting conditions, thereby weakening class discriminability. To address these issues, we propose a frequency-guided cross-modal distillation framework with illumination-invariant prototype alignment for RGB-IR object detection. The proposed Object-level Frequency Expert Distillation (OFED) decomposes Region of Interest (RoI)-level fused features into low-, mid-, and high-frequency bands using Fast Fourier Transform (FFT). For each band, lightweight learnable modality gates adaptively combine RGB and IR teacher signals, enabling the fused representation to receive band-selective and modality-specific distillation. In addition, Illumination-invariant Prototype Alignment (IPA) aligns Class Activation Mapping (CAM)-based object-level fused features with class-specific prototypes, reducing illumination-induced intra-class variance and improving class-level separability. Both OFED and IPA are used only during training and are removed at inference time. Experiments on FLIR, LLVIP, and DroneVehicle demonstrate that the proposed framework consistently improves over M2D-LIF, achieving gains of +2.1 mAP on FLIR, +1.0 mAP on LLVIP, and +1.6 mAP on DroneVehicle, while introducing no additional parameters or computational overhead during inference. These results show that frequency-aware cross-modal distillation and illumination-invariant prototype alignment provide an effective and practical training strategy for robust RGB-IR object detection.