An explainable multimodal framework for exploring depression-associated speech representations in Alzheimer's disease

Abstract

Depression in Alzheimer's disease (AD) is clinically challenging to characterise because affective and cognitive manifestations frequently overlap with the underlying neurodegenerative condition. This study investigates depression-related speech patterns in AD using an explainable multimodal framework that integrates linguistic and acoustic features from elicited picture-description speech. To address the absence of directly observed depression labels in the target AD cohort, we employ a source-informed cross-dataset transfer strategy. Unsupervised clustering identified two distinct speech sub-phenotypes within the AD cohort. By comparing these unsupervised representations to a depression-labelled source corpus, one cluster demonstrated mathematical proximity (measured via cosine similarity in a shared latent space) to the source depression centroid. We explicitly acknowledge that this proximity reflects geometric similarity in a shared latent space, which may encapsulate overlapping domain characteristics alongside affective variance. To validate the structural integrity and separability of these discovered clusters, a proxy supervised classification task was employed. The retained multimodal artificial neural network (ANN) integrated information from both modalities and achieved 92.73% cluster separability accuracy. This demonstrates that the representation-derived two-cluster partition was highly learnable on the independent internal hold-out test set; it does not establish clinical validity, longitudinal stability, or diagnostic validity of the identified sub-phenotypes. For interpretability, Shapley Additive exPlanations (SHAP) analysis was utilised to extract feature attributions and organise influential acoustic and linguistic features into a multimodal taxonomy. Furthermore, a Multimodal SHAP Attribution Magnitude (MSAM) score was derived as an exploratory model-based attribution score rather than a clinically validated severity measure. Ultimately, this study provides an interpretable computational baseline for characterising depression-related speech variance in AD, establishing a structural foundation for future prospective validation using independently labelled clinical cohorts.