Generative AI-driven sensor fusion for missing-modality and open-set smartphone activity recognition

Abstract

Smartphone-based Human Activity Recognition (HAR) is increasingly used in Internet of Things (IoT) and edge-intelligence applications for health monitoring, context-aware services, and mobile sensing. However, many existing HAR systems assume complete sensor availability and closed-set activity classification, assumptions that may not hold in real-world deployments where inertial sensor streams can become unavailable and users may perform previously unseen activities. This study proposes a lightweight multimodal HAR framework that jointly addresses missing inertial modalities and open-set activity recognition in smartphone-based IoT edge environments. The proposed approach employs a bidirectional Cross-Modal Conditional Generative Adversarial Network (CM-CGAN) to reconstruct a missing accelerometer or gyroscope stream from the available modality, followed by multimodal feature extraction and Random Forest classification with a calibrated maximum-posterior rejection rule for unknown-activity detection. Experiments conducted on the public UCI-HAR dataset using a subject-disjoint protocol and repeated-run evaluation demonstrate that synthesis-based multimodal fusion substantially improves recognition performance compared with noisy replacement strategies and remains close to the fully observed multimodal setting. Signal-similarity analysis further shows strong agreement between synthesized and real inertial streams in both temporal and spectral domains, while runtime measurements indicate that the complete reconstruction-and-classification pipeline remains compatible with real-time smartphone and edge-device deployment. Overall, the findings suggest that cross-modal generative reconstruction can improve the robustness and reliability of intelligent IoT sensing systems operating under incomplete and uncertain sensing conditions.