Dynamics-Guided Cross-Modality Masked State Space Modeling for Robust Wearable Human Activity Recognition

Abstract

Wearable sensor-based Human Activity Recognition (HAR) is important for continuous health monitoring, assistive systems, and context-aware systems. Despite recent progress, real-world deployment remains difficult due to issues such as sensor noise, missing data, device heterogeneity, and limited annotated samples. Although self-supervised learning (SSL) methods help reduce dependence on manual labeling, many existing approaches still employ random masking schemes that do not effectively focus on informative motion patterns and dynamic activity transitions. To address these limitations, this work proposes a dynamics-guided cross-modality masked self-supervised framework for robust wearable HAR. The proposed approach introduces motion-aware masking using kinematic interaction-aware features to prioritize high-dynamics temporal regions during reconstruction. A cross-modality reconstruction objective enables complementary sensor streams to recover partially missing information during masked reconstruction. The framework employs a bidirectional selective state space encoder for efficient long-range temporal modeling with linear computational complexity. Experimental results demonstrate competitive performance, achieving 91.48 ± 0.09% accuracy and 91.55 ± 0.09% Macro F1 on UCI HAR across three independent random seeds, along with 79.01% accuracy and 81.17% Macro F1 on PAMAP2. Additional robustness experiments under simulated sensor degradation show stable performance under missing sensor conditions. These results suggest that dynamics-guided masking and cross-modality state space modeling provide an effective framework for robust wearable HAR under noisy sensing conditions.