A Deep Multiple Instance Multiple Label Learning Framework for Smartphone-Based Human Activity Recognition

Abstract

Human Activity Recognition through smartphones plays a crucial role in several medical states of affairs such as patient monitoring, elder-care, and post-surgery recovery. Most of these works require precisely labelled smartphone accelerometer data for training supervised learning classifiers. However, precise data annotation is a difficult and laborious task that prevents real-life adaptability of such solutions. Interestingly, semi-supervised learning of the accelerometer instances along with grossly labelled data could indicate a precise way of predicting constituent activities (such as standing, walking) from a composite daily activity (such as cooking). Most state-of-the-art works consider either the data to be precisely annotated or the detection of composite daily activities as an atomic activity class. However, a closer look at the recognition of the constituent sub-activity parts can provide better context awareness and predict the reason behind any inability to perform a composite activity. To address this challenge, we formulate composite activity recognition as a Multiple Instance Multiple Label (MIML) learning problem, where grossly labelled bags are used instead of precisely annotated instances. This enables the prediction of constituent activities within a composite activity despite labelling uncertainty. The proposed framework is implemented using various deep architectures, namely MIML-CNN, MIML-LSTM, MIML-ResNet, and MIML-AlexNet. The proposed framework takes grossly labelled data in the form of bags of multiple instances, labelled with possible constituent activities, which are then fed into the respective deep networks for activity classification and sub-activity prediction. The proposed models are evaluated on a real-world smartphone accelerometer dataset collected from 20 users. Among these, the MIML-ResNet achieved the highest accuracy of 98.16% for activity classification in the presence of labelling errors. The results demonstrate the effectiveness of the proposed MIML approach for recognizing individual activities from grossly labelled composite activity data.