Jul 2026· Applied and Computational Engineering· Vol 243, pp. 235-242· 0 citations
TL;DR
The paradigm shift to "vehicle-road synergy" infrastructure as a critical mechanism for providing the safety redundancy and edge-computing capabilities needed for fully reliable, next-generation autonomous driving systems is explored.
Abstract
This paper presents a comprehensive survey on deep learning based multi-modal perception fusion frameworks for autonomous driving systems. The traditional single-modal solutions have intrinsic limitations in robustness against harsh weather and dynamic environments, and the integration of complementary sensors, such as cameras, LiDAR, and millimeter-wave radar, provides a critical pathway to high-level automation. We provide a systematic review of the basic fusion paradigms, classifying them into early, late and deep (feature-level) architectures, and evaluating their trade-offs in terms of computational overhead, architectural modularity and joint feature learning. Special focus is given to state-of-the-art architectures, comparing the high representational power of the Tensor Fusion Networks (TFN) to the computationally-efficient Multi-modal Circulant Fusion (MCF) and context-aware Transformer frameworks. This study places real-world deployment challenges in the context of algorithmic paradigms, investigating the friction between “black-box” deep learning models and stringent functions. Besides algorithmic paradigms, this work puts real-world deployment challenges into context and discusses the friction between ``black-box'' deep learning models and rigorous functional safety standards (e.g., ISO 26262), and energy constraints of new energy vehicles. Finally, we explore the paradigm shift to "vehicle-road synergy" (V2X) infrastructure as a critical mechanism for providing the safety redundancy and edge-computing capabilities needed for fully reliable, next-generation autonomous driving.
Interaction with the physical world differentiates physical AI from other forms of AI. Autonomous driving exemplifies this; vehicles must perceive and respond to dynamic environments with human-like or better perception-reaction times. This survey addresses the fundamental challenge of deploying high-performance models for multimodal fusion in resource-constrained automotive environments. We organise state-of-the-art deep learning approaches into five paradigms—CNN-based, transformer-based, dense BEV-based, sparse-based, and hybrid—revealing trade-offs in accuracy, latency, and efficiency, as well as strengths and limitations in robustness under adverse operational design domains. The hardware-aware perspective is a differentiating contribution, presenting strategies for deployment on automotive platforms, reducing inference latency by up to 50% and improving robustness in adverse conditions by up to 20%. By synthesising sensor fusion, deep learning, compute platforms, and hardware-awareness, this work equips researchers and practitioners with actionable insights and strategies for perception systems, bridging theoretical advances and production-grade autonomous driving requirements.
Ken Power, M. Halton, Ciarán Eising· IEEE Open Journal of Vehicul...· 0 citations
A systematic analysis of the machine learning and deep learning models underpinning vehicle autonomy, spanning classical convolutional neural networks for object detection and semantic segmentation to recurrent and Transformer-based architectures for trajectory prediction and motion planning is presented.
Esraa Khatab, Fares Fathy, Abdallah AlKholy et al.· Machine Learning and Knowled...· 0 citations
: Visual simultaneous localization and mapping (VSLAM) is a key technology for mobile robotics, autonomous driving, and embodied intelligence, enabling self-localization, environment reconstruction, and scene understanding. Although conventional geometric methods have achieved notable success, their performance often degrades in challenging conditions, such as low-texture scenes, severe illumination changes, dynamic interference, and long-term environmental variations. Recent advances in deep learning have created new opportunities to improve VSLAM through stronger feature representations, learned priors, semantic perception, and emerging map representations. At the same time, the increasing adoption of learning-based modules has raised important questions about integration strategies, generalization, interpretability, and real-time deployment. This paper presents a systematic review of deep learning-enhanced VSLAM, with a particular focus on how learning models are incorporated into classical simultaneous localization and mapping (SLAM) pipelines and how they function within the overall system. To provide a unified perspective, existing methods are organized into five categories according to their fusion interfaces with geometric SLAM pipelines: observation-level interfaces, constraint/prior/weight-level interfaces, solver-level interfaces, representation-level interfaces, and system-level integration interfaces. Based on this taxonomy, representative approaches are comparatively analyzed for accuracy, robustness, efficiency, and deployability. In addition, this review summarizes common design principles, including geometric consistency constraints, error propagation characteristics, and typical failure modes, and further discusses open challenges and future directions such as lightweight deployment, cross-domain adaptation, dynamic map modeling, and long-term consistency maintenance. This review aims to provide a structured reference for the analysis, design, and deployment of learning-enhanced VSLAM systems.
Xiruo Chen, Ouyang Qi, Sihong Meng et al.· Computers, Materials & C...· 0 citations
In the field of end-to-end autonomous driving, extreme visual degradation environments, such as nighttime and low-light conditions, often lead to a sharp decline in sensor perception capabilities, thereby triggering safety risks like trajectory deviation. Under such scenarios, significant feature distribution uncertainty and spatial divergence arise between the camera and LiDAR modalities. Traditional closed-box fusion mechanisms, lacking explicit physical constraints, are highly susceptible to failure in Out-of-Distribution (OOD) scenarios. To address this issue, this paper proposes DynaFuser, an uncertainty-aware multimodal fusion architecture designed for dynamic and complex environments. The architecture introduces a Collaborative Dual-stream Adaptive Feature Refinement Mechanism (CDA-FRM) to perform direction-aware enhancement on visual textures and LiDAR geometric structures, respectively. The core contribution lies in proposing a Heterogeneous Modal Dynamic Entropy Weight (HM-DEW) fusion strategy based on non-extensive statistical mechanics. This strategy utilizes variable-order Tsallis entropy to quantify intra-modal uncertainty in real-time and employs Kullback-Leibler (KL) divergence to achieve cross-modal consistency alignment. Furthermore, to address trajectory deviation caused by perceptual perturbations, the model constructs a Dynamic Channel Decoupling Waypoint Prediction Mechanism (DCD-WPM), which integrates asymmetric feature reconstruction with a spatio-temporal gated autoregressive decoder to significantly improve the geometric consistency and temporal coherence of trajectory outputs. Evaluated in the CARLA closed-loop simulation environment, DynaFuser achieves state-of-the-art performance on both the Longest6 and LAV benchmarks, obtaining a Driving Score (DS) of 60.45% and 67.42%, respectively, along with a Route Completion (RC) of 94.33% and 93.24%, thereby demonstrating superior scene adaptability and generalization capability. Note to Practitioners—Autonomous driving in real-world traffic remains highly challenging due to dynamic lighting conditions, unpredictable obstacles, and sensor degradation under adverse environments. In practice, engineers frequently encounter challenges when fusing data from cameras and LiDAR due to their distinct characteristics and occasional misalignments, which can lead to unsafe trajectory predictions. This work introduces DynaFuser, a dynamic multimodal fusion framework designed to improve trajectory planning reliability in complex environments. The core concept involves adaptively refining camera and LiDAR features separately, subsequently combining them through a dynamic weighting strategy that accounts for environmental conditions, such as low illumination or dense traffic. By doing so, the system significantly reduces trajectory errors and improves route completion in challenging scenarios, as demonstrated in CARLA closed-loop benchmarks. For industry practitioners, this approach provides a more robust trajectory planning baseline, particularly for autonomous driving fleets operating in urban areas where lighting and occlusion frequently vary. Our current validation is limited to simulation, and real-world deployment will require further testing with large-scale on-road datasets. Moreover, computational efficiency remains a critical factor when integrating such models into automotive-grade processors. Future work may extend this framework to incorporate additional modalities (e.g., radar, V2X communication) and further optimize inference speed. Beyond autonomous vehicles, the proposed fusion strategy could also benefit other robotic applications requiring reliable navigation under uncertain sensor conditions.
Hongxin Xue, Qiqi Jin, Luyang Li et al.· IEEE Transactions on Automat...· 0 citations
Reliable autonomous driving perception remains difficult in adverse weather because camera appearance, LiDAR point density, and radar responses degrade in different and condition-dependent ways. Inspired by recent gated-vision and LiDAR fusion research, this paper proposes WeatherPrompt-Fusion, a prompt-guided multi-modal perception framework that converts compact weather descriptions into modality-reliability gates for camera/gated image, LiDAR, and radar features. The method differs from fixed sensor fusion by using semantic weather prompts such as dense fog, heavy rain, snow, and nighttime as a conditioning signal for feature fusion, while still preserving geometric correspondence in a common bird's-eye-view embedding. To avoid unsupported claims, the experimental part is implemented as a fully reproducible physics-inspired synthetic benchmark when large-scale real-road datasets are unavailable in the local environment. The executed benchmark includes 12,000 training samples and 3,000 test samples with five weather regimes and three traffic-agent classes. WeatherPrompt-Fusion obtains a macro mAP of 0.752, improving over fixed average fusion (0.673), single-modality LiDAR (0.621), radar (0.604), and camera-only perception (0.504). Under fog, the proposed model reaches 0.744 mAP versus 0.651 for fixed fusion and 0.702 for naive concatenation. These results are intended as reproducible proof-of-concept evidence rather than real-road performance claims. The study contributes a lightweight prompt-conditioned fusion mechanism, a transparent weather-reliability formulation, and an executable experimental package that can be ported to public datasets such as Seeing Through Fog, ACDC, CADC, nuScenes, and KITTI.
L. Mansour· International Journal of Adv...· 0 citations
Abstract. Remote sensing acquisition systems rely on a range of platforms, from drones to satellite missions, to record multimodal Earth surface data. This fact encourages the preparation of datasets with complementary properties, thereby increasing their discriminative potential. A common complementary combination is between Hyperspectral and LiDAR-generated digital surface model data. While engaging, this fusion poses challenges for specific applications. Multiple works fuse these modalities at the feature level using vector concatenation, maximization, or averaging. Although functional, these methods omit target interactions between the modalities. Another challenge in remote sensing is the quantity and quality of labels required by deep learning methods, which are expensive, error-prone, and difficult to scale. We address the challenges above by proposing a self-supervised processing framework based on cross-modal attention that effectively fuses features at multiple levels, thereby exploiting complementary information across data streams. Specifically, our method is founded on a pseudo-Siamese network that reweights each modality’s features with information from the other via a mirrored cross-modal attention. The network’s objective is to maximize the similarity between the feature representations of both streams. A fusion network builds a latent representation using the learned encoders and attention modules. Then, a k-Nearest Neighbor classifier categorizes each sample within the representation using ten labels per class. Our experiments show that our spatial- and channel-spatial cross-modal attention approaches outperform well-established fusion methods for label-efficient land cover classification across datasets. Our findings lay the groundwork for fusion methods that effectively exploit inter-stream data relationships to encourage complementarity.
Jonathan González Santiago, Wolfgang Gross, K. Schulz et al.· ISPRS Annals of the Photogra...· 0 citations