Multimodal Deep Learning Driven English-speaking Emotion Recognition and Adaptive Teaching Strategy Generation

Abstract

It is critical for emotion-aware multichannel adaptation techniques to have high accuracy in terms of recognizing emotions, particularly in English-speaking learners' cases. Recurrent neural network-based methods, aggregation-based approaches, and conventional multimodal fusion techniques are known to have some drawbacks related to time dependency, dynamicity of emotional transitions, and inter-speaker variance robustness. In an attempt to address these limitations, we propose a task-specific multimodal framework that employs an emotion-structured transformer encoder coupled with a progressive emotion distillation strategy. What should be noted first about this paper is that its contribution lies in neither technique employed, since they were previously utilized in different works. The key idea behind the proposed approach is the combination of existing emotion recognition techniques into a multimodal pipeline that facilitates improved emotion recognition and robust representation learning. Estimation of the speaker-independent representation, emotion-aware representation, and improved emotion recognition are achieved through employing emotion-prior masked attention, emotion-gated feature transformation, multi-head attention pooling, and boundary-aware auxiliary supervision in ESTE. Further improvement in training the recognition model is obtained by incorporating emotionally ambiguous samples progressively and a distillation process based on soft labels, in addition to auxiliary tasks such as valence-arousal regression and boundary detection. Improved results are empirically observed when compared with selected baselines using emotion recognition benchmark datasets. As a conclusion, an interpretative teaching strategy generation method relying on the rules of teaching guidelines is suggested.