Multi-Attribute Scene Context and Pedestrian Behaviour Recognition using PEFT-Tuned Vision Transformer Model for Autonomous Driving System

Abstract

Understanding the real-time behaviour of pedestrians and their surrounding context is essential for road safety. Unfortunately, current perception systems struggle to capture how multiple cues jointly influence behavior in diverse traffic scenarios. Most existing models in Intelligent Transportation Systems (ITS) and Autonomous driving system (ADS) predict only a single attribute, such as intent or trajectory, while overlooking the interaction between behavioral and environmental factors. We propose a unified multi-attribute framework based on a Vision Transformer, which is enhanced with lightweight, parameter-efficient adapters. This innovative modular design enables independent learning of eight complementary attributes, including pedestrian behavior (action, attention, crossing, occlusion) and scene context (weather, time of day, road presence, density). Our proposed framework updates fewer than 2% of the model parameters. We ran our proposed framework using the JAAD dataset, and the framework achieves over 90% accuracy on most attributes and exhibits consistent behavior–context relationships and demonstrates robustness under varied environmental conditions. To evaluate real-time suitability, we implemented an adaptive strategy that activates only the most relevant adapters based on scene conditions. This approach reduces computational cost by 28.5% and increases throughput by 40.8% without loss of accuracy. This research distinctly highlights the practical potential of modular, PEFT-enhanced transformers for scalable multi-attribute perception as foundational vision models continue to advance. The best-performing adapters and inference code are available on Github Repo.