MM-PSYCHE: Multimodal Multitask Psychological Characteristic Estimation Through Cross-Domain Semi-Supervised Learning

Abstract

Psychological characteristic estimation from multimodal in-the-wild behavior is usually studied using separate corpora, each annotated for a single target task. Such annotation fragmentation limits cross-task learning and cross-domain generalization across affective, dispositional, and interactional phenomena. To address this problem, we use emotion, apparent personality trait, and ambivalence recognition as representative tasks and introduce MM-PSYCHE, a framework that unifies heterogeneous single-task corpora through cross-domain semi-supervised learning (SSL) with GradNorm-based adaptive task weighting. The framework jointly trains models for emotion, apparent personality trait, and ambivalence recognition. Its multimodal fusion model contains three task-aware components: 1) Task-Specific Projectors, which produce task-conditioned prediction embeddings and re-embed them into a hidden space; 2) Task-Specific Query-Based Cross-Attention Fusion, which uses the prediction embeddings as task-dependent queries and shared modality embeddings as keys and values; and 3) Guide Banks, which are learnable task-specific prototypes that anchor the final predictions. Experiments on the task-specific corpora show competitive supervised learning (SL) performance and more pronounced gains in cross-domain SSL, where heterogeneous corpora can be exploited without joint annotations. On the unseen Multimodal EmotionLines Dataset (MELD), multitask SSL improves generalization in emotion recognition, increasing the macro-F1 score by 7.5 percentage points (35.0% vs. 27.5%) over single-task SSL. The results indicate that cross-domain SSL is useful for fragmented multimodal corpora with heterogeneous psychological annotations, while also showing why single-task SL, single-task SSL, and multitask SSL should be evaluated separately. The source code is publicly available at https://github.com/LEYA-HSE/MM-PSYCHE.