Trends and Advances in Learning-Based Emotion Recognition Across Heterogeneous Data Modalities

Abstract

Emotion recognition has emerged as a critical component in the development of intelligent systems capable of understanding and responding to human affective states across applications such as healthcare, human-computer interaction and smart environments. Despite substantial progress, achieving accurate and generalizable emotion recognition remains challenging due to the complex, subjective and multimodal nature of human emotions. This review presents a comparative analysis of recent studies employing machine learning, deep learning, hybrid and ensemble approaches across diverse data modalities, including speech, facial expressions and physiological signals. The analysis highlights that deep learning and hybrid models significantly enhance feature representation by capturing spatial and temporal dependencies, while ensemble techniques improve classification robustness and stability. Furthermore, multimodal approaches consistently outperform unimodal systems by integrating complementary emotional cues from heterogeneous data sources. However, several limitations persist, including small and imbalanced datasets, limited generalization across diverse populations, noise in physiological signals and high computational complexity that restricts realtime deployment. Based on these observations, this study identifies key research gaps and emphasizes the need for scalable, lightweight and generalized multimodal emotion recognition frameworks. Future research should incorporate multimodal fusion framework that effectively integrate heterogeneous data modalities, robust feature learning strategies and computationally efficient architectures with improved accuracy, adaptability and practical applicability.