Lightweight cross-modal RGB-IR fusion network for real-time aerial surveillance
Abstract
Real-time onboard object detection is essential for autonomous UAV applications requiring rapid and reliable decision-making in dynamic environments. Most existing onboard detectors rely on RGB imagery due to its computational efficiency; however, RGB-based models suffer significant performance degradation under low-illumination conditions. Although recent RGB–infrared (RGB–IR) fusion approaches enhance detection by leveraging complementary thermal information, they often employ complex architectures that introduce feature redundancy, and high computational overhead, limiting their suitability for real-time edge deployment. To address these limitations, this paper proposes a lightweight RGB–IR object detection framework for real-time onboard aerial applications. A guided difference-based fusion mechanism is introduced to adaptively balance RGB and IR contributions while reducing cross-modal redundancy. Additionally, a lightweight variant of the FasterNet module is developed to improve feature extraction efficiency, particularly within the backbone, which dominates inference cost. The results demonstrate its efficiency on three aerial multimodal datasets, DroneVehicle, VEDAI and RGBT-Tiny for small objects for surveillance applications. The proposed model achieves 93.0% mAP50 on VEDAI and 78% mAP50 on DroneVehicle. With only 2.74M parameters, it runs at 2.9 ms per image on an RTX 4080 and achieves 35.71 FPS on Jetson Orin AGX and 26.3 FPS on Jetson Orin Nano, with 5.6 MB network size, validating its effectiveness for edge UAV deployment.