AR-Based System for Shoulder-Joint Rehabilitation with Markerless Measurement of Ranges of Motion and a Recommendation Module for Correction of Therapeutic Images

Abstract

The full-scale war in Ukraine has caused a sharp rise in the number of patients with musculoskeletal injuries who need prolonged upper-limb rehabilitation. Traditional physicaltherapy methods have several limitations: the subjective character of range-of-motion (ROM) measurement with a manual goniometer, the lack of objective progress monitoring between in-person visits to a specialist, and the low engagement of patients caused by the monotony of exercises. This paper presents an augmented-reality (AR) system for shoulder-joint rehabilitation that combines three key components: an AR arttherapy environment based on the Magic Leap 2 headset that motivates the patient to perform therapeutic movements in the form of “drawing in space”; a markerless module for measuring three shoulder-joint angles (flexion 160-180°, extension 50-60°, abduction 180°) based on three-camera video analysis using MediaPipe Pose Landmarker and an adaptive Kalman filter with an innovation-gate mechanism; a recommendation module that, on the basis of predictive mathematical models of recovery dynamics built from data of three patients, personalises the therapeutic images with the help of large language models (LLM) and a generative image model. Preliminary clinical validation on two patients who underwent rehabilitation with the recommendation module showed a positive ROM recovery dynamic compared with the typical predicted trajectory. The obtained results confirm the practical suitability of the AR system for objective and personalised rehabilitation.