AI-enabled Extended Reality System for Real-Time Driver Behavior Monitoring

Abstract

The World Health Organization (WHO) reports that approximately 1.19 million people die annually due to road traffic accidents, while 20 to 50 million others suffer non-fatal injuries, often leading to disabilities. One of the primary causes is driver distraction, with mobile phone use being an increasing concern for road safety. This study utilized the image-based distracted driver dataset V2, featuring data from 44 drivers across 10 different classes. The dataset, in RGB format, includes images captured from a side-view camera focusing on the driver’s body. To enhance the dataset, a new class was added for drowsy drivers, as statistics indicate that drowsy driving contributed to an estimated 17.6% of fatal car crashes between 2017 and 2021. Transfer learning techniques were applied for classification purposes, and the proposed Deep Convolutional Neural Network (DCNN) model, based on MobileNet, achieved 93.60% accuracy and an F1-score of 0.94. The trained AI model was subsequently deployed on an Extended Reality (XR) device to facilitate real-time monitoring of driver behavior.