Multimodal Depression Detection Varies with Cross-Dataset Differences

Abstract

Past work in multimodal depression detection has focused almost entirely on a single dataset with limited socioeconomic diversity and lacking in clinically validated depression diagnoses. To address these limitations in multimodal depression detection, we used clinically-validated depression diagnoses in two distinct datasets that differed by socioeconomic status (SES): A problem-solving interaction and a clinical interview. Participants in the former were of low SES; participants in the latter were of low- to middle SES status. Both included members of racial minorities. Because findings may vary with choice of modeling approach, we studied both a classical machine learning approach and two deep-learning approaches. The former consisted of handcrafted features with the Support Vector Machine classifier; the latter two were deep approaches with learnt features. One was a baseline deep approach that used a multimodal transformer with pairwise crossmodal attention. The other was our proposed multimodal transformer that used the novel Learnable Multimodal Query Crossmodal Attention (Learnable Multimodal Query CA). For each, we assessed accuracy, demographic fairness, and cross-dataset generalizability at depression detection. For both datasets, optimal embedding representations varied. The accuracy of the proposed multimodal transformer with the Learnable Multimodal Query CA and the classical approach was relatively higher. In the clinical interview, accuracy was lower for the baseline deep multimodal transformer. Generalizability across datasets was lower for both classical and deep approaches.