TruckAct: A multi-source, multimodal dataset for real-world truck driver activity recognition.

Abstract

Despite the significant hazards posed by distracted driver behaviors, robust monitoring of truck driver activities remains constrained by the scarcity of publicly accessible, high-quality datasets. To fill this gap, we introduce TruckAct, the first naturalistic, multi-source, and multimodal dataset dedicated to truck driver activity recognition. Collected from nine professional drivers under real-world trucking operations without any artificial intervention, TruckAct contains approximately 67 h of synchronized onboard videos, vehicle telemetry, and wearable wristband signals, partitioned into 24,000 non-overlapping 10-second clips with nine well-annotated activity classes. Built on TruckAct, we propose TruckAct-Net, a multi-stream, hierarchical framework that consists of three core components: a Frequency-domain Query Gating (FreqQG) module that suppresses vibration-induced noise in in-cabin video sequences, a dual-stream Time-Frequency TransFormer (TF-Former) encoder that jointly models the temporal evolutions and spectral patterns of vehicle dynamics and wristband signals, and a reliability-guided adaptive fusion strategy that adaptively adjusts the contribution of each modality according to its reliability. Extensive experiments demonstrate that TruckAct-Net achieves robust performance and maintains strong stability even when specific modalities are noisy, occluded, or missing. Visual input delivers discriminative cues, while vehicle motion signals and wristband data help resolve ambiguities and stabilize prediction results. The dataset and source codes are available at https://github.com/WangQF1/TruckAct.