Multimodal GenAI for Scalable Synthetic Dataset Generation in Autonomous Vehicles: Advancements, Challenges, and Opportunities in Intelligent Transportation Systems [Feature Article: Generative AI for Synthetic Dataset Generation]

Abstract

Generative artificial intelligence (GenAI) is emerging as a transformative tool for synthetic dataset generation in advanced driver-assistance systems (ADASs). Traditional autonomous vehicle (AV) datasets face high collection costs, limited edge-case diversity, and scalability challenges, making alternative data generation approaches essential. This article introduces an innovative application of multimodal GenAI models for AV perception tasks with a focus on ADAS applications. Specifically, the study explores text-to-image, text-to-video, image-to-video, and image-to-image models to generate high-fidelity data for object detection, segmentation, and trajectory prediction. Despite their promise, practical deployment is obstructed by prompt adherence failures, mode collapse, uncontrollability, domain adaptation inconsistencies, and computational inefficiencies. We provide a comparative evaluation of these models, benchmarking their realism, consistency, and adaptability in bridging the synthetic-to-real gap and enhancing AV robustness. Experimental results highlight the strengths and limitations of different approaches, with insights into controllable data synthesis and improvements in real-time efficiency. Furthermore, YOLO models fine-tuned on GenAI-generated datasets achieve superior detection performance, with GAI_YOLOv7 reaching 74.48% mean average precision (mAP) and GAI_YOLOv8x achieving 76.16% mAP, outperforming virtual reality-based models (29.76%, 23.44%).