A time-aware multilingual multimodal framework for detecting depression-related expressions on social media
TL;DR
This study introduces a time-aware multilingual multimodal framework for detecting depression-related expressions in social media posts collected from X (formerly Twitter), and provides a systematic evaluation of encoding and fusion strategies.
Abstract
Depression remains one of the most prevalent mental health conditions worldwide, yet most automated detection systems focus on English text and overlook the multilingual, multimodal nature of social media communication. This limitation is particularly significant in India, where users frequently write using Hinglish and other Hindi variants, posing substantial challenges for standard language processing pipelines. This study introduces a time-aware multilingual multimodal framework for detecting depression-related expressions in social media posts collected from X (formerly Twitter). The model jointly learns from text, images, and posting time, enabling it to capture how users express themselves across languages and modalities. To support this work, we construct and annotate a dataset of 15,739 posts from 3,471 users covering four linguistic varieties: English, Roman Hindi, Devanagari Hindi, and Hinglish. The contributions of this work are twofold: it presents a multilingual multimodal dataset for Indian social media depression-related content detection, and it provides a systematic evaluation of encoding and fusion strategies, achieving an F1-score of 0.782 with cross-lingual transfer capability demonstrated on a Bengali corpus. All outputs are intended as a content-screening aid and do not constitute clinical diagnosis.