Reliability, validity, and usability of an 18-item immersive virtual reality action research arm test: a validation study in stroke survivors

Abstract

Background Stroke is a leading cause of adult disability, with upper extremity hemiparesis affecting approximately 80% of survivors. While the Action Research Arm Test (ARAT) is the reference standard for assessing motor function, its clinical use is hindered by being resource-intensive and requiring standardized physical props. Virtual Reality (VR) has emerged as a promising method for automated and objective assessment. Objective This study aimed to develop and validate an 18-item immersive VR version of the ARAT (ARAT-VR) and assess its validity, reliability, and usability among stroke survivors, healthcare professionals, and healthy controls. Methods Thirty stroke survivors, 30 healthy controls, and eight healthcare professionals were recruited. The ARAT-VR system, developed for the PICO 4 Ultra Enterprise headset, utilizes an iToF depth sensor for controller-free optical hand tracking and retains 18 of the original 19 items. Content validity was evaluated by professionals. Concurrent validity was measured via Spearman's correlation between ARAT-VR and traditional ARAT scores. Test-retest reliability (Intraclass Correlation Coefficient, ICC) was assessed over a 7-day interval. Usability was measured using the System Usability Scale (SUS). Results Regarding content validity, most of the 18 ARAT-VR items were rated as comparable to physical tasks specifically in visual-motor control demands (Median = 0), except for manipulating minute objects (e.g., 6 mm marbles), which was rated as more difficult (1.0 [1.0–1.0]). Regarding concurrent validity, the ARAT-VR scores for the paretic hand strongly correlated with both the 18-item traditional ARAT (rs = 0.845, 95% CI: 0.673–0.965, p < 0.001) and the full 19-item ARAT (rs = 0.845, 95% CI: 0.696–0.961, p < 0.001). Bland-Altman analysis indicated a mean bias of 1.77 points with 95% limits of agreement from −10.07 to 13.60. Test-retest reliability was excellent (ICC = 0.98, 95% CI: 0.96–0.99, p < 0.001), with a Minimal Detectable Change (MDC) of 5.36 points, representing the minimum change required to exceed measurement error. Usability was rated as excellent (Median SUS = 85.0 for healthy controls and 81.3 for stroke survivors; p = 0.212). Conclusions Initial findings suggest that the 18-item ARAT-VR possesses favorable concurrent validity, high test-retest reliability, and acceptable usability, highlighting its potential as a supplementary digital assessment tool for stroke survivors. The calculated MDC of 5.36 points offers a preliminary reference for future longitudinal monitoring. However, its readiness for routine clinical implementation remains unestablished. Further large-scale studies are necessary to validate the system across broader populations and explore hardware optimizations, such as the integration of haptic feedback.