Can Vision-Language Models Extract Relevant Multimodal Cues for Post-Inference Cue Integration? A Comparison of Affective Judgments on Adult and Child Interaction Data

Abstract

Inferring human affective states from multimodal behavioral cues is essential for socially aware AI and human-robot interaction. Conventional research practice either relies on hand-crafted behavioral detectors or end-to-end deep learning models that often lack interpretability and contextual adaptation. Recent work suggests that vision-language models (VLMs) may serve as informative priors for affective inference; however, their ability to extract and reason over behavioral cues in naturalistic human-human interactions remains unclear. We propose EPIC, a three-stage framework to delineate VLMs’ affective inference into testable sub-processes: multi-modal cue extraction, cue-level affect prediction, and cue integration. Applying the framework, we conduct a case study with one VLM on affective inference tasks with a real-world dyadic interaction dataset. Preliminary results show that explicit cue integration improves performance when cue quality is high, reaching up to a 14% gain. However, VLM’s cue extraction shows strong visual bias and underuses speech and paralinguistic information, leading to weaker performance when inferring arousal and children’s states. Overall, our results suggest VLM might provide promising interpretable priors for affective reasoning but remain limited in modality balance, context sensitivity, and demographic robustness.