Toward Reliable Autonomous Systems in Fog: A Multi-Task Learning Framework for Joint Image Restoration and Object Detection

Abstract

Foggy weather poses a significant challenge for object detection in autonomous vehicles due to degraded visibility. To improve the visual quality, existing methods make efforts to jointly learn image restoration and object detection tasks. However, training a restoration subnet requires pixel-corresponding image pairs, which are unavailable in real-world scenarios due to scene changes. To address this challenge, we design a novel Dual-stream Learning Framework (DLF) that incorporates a Zero-referenced Image Dehazing subnet (ZIDnet) into a detector. By designing image restoration losses based on the atmosphere scattering model, ZIDnet can work in an unsupervised manner. This can significantly expand the applicability of multi-task learning from synthetic image pairs to real-world scenarios where only hazy images are available. Additionally, a feature selection module is newly introduced to emphasize informative features while suppressing noisy responses under foggy conditions. Furthermore, to mitigate conflicts between image restoration and object detection losses during joint training, we introduce a task balancing method to automatically adjust the contributions of image restoration and object detection tasks. Experiments on the Foggy-Cityscapes benchmark (synthesized from real driving scenes) and the real-world RTTS dataset, collected from actual autonomous vehicles, demonstrate our method’s superior performance in practical foggy conditions. Physical experiments conducted on an autonomous driving platform further demonstrate the generalization capability of the proposed method across various foggy conditions. Source codes and pretrained weights are available at https://github.com/Newj596/DLF-Detect. Note to Practitioners—In autonomous driving systems, foggy weather can severely degrade camera image quality and undermine object detection performance. A common engineering practice is to adopt a cascaded pipeline that “dehazes first, then detects”. However, this two-stage approach faces some major deployment challenges: 1) it is extremely difficult to collect pixel-aligned foggy/clear image pairs from the same scene and viewpoint for supervised training; and 2) models trained solely on synthetic fog images often fail to generalize to real-world road conditions. To address these issues, this work proposes an end-to-end dual-stream-based learning framework, in which an unsupervised loss derived from an atmospheric scattering model is used to jointly train dehazing and detection branches. As a result, only real foggy images from driving scenarios are required and no paired clear images are needed. Our approach eliminates the need for expensive paired-data acquisition, i.e., it enables engineers to leverage foggy images already collected by existing vehicle fleets and continuously improve system performance.