Multimodal Subjective Intention Modeling in Social Interactions

Abstract

Understanding human intentions in complex social interactions is essential for developing socially intelligent systems, yet remains challenging due to their subjective, context-dependent, and multi-interpretable nature. Existing approaches typically reduce intention estimation to single-label outcome based prediction tasks, overlooking the diversity of subjective interpretations and the causal mechanisms that shape them. In this work, we propose a shift toward modeling intention as a distribution over multiple narrative interpretations grounded in multimodal social cues and observer-dependent factors. Building on theories of social perception, situation modeling, and causal representation learning, I aim to develop a causal generative model that disentangles contextual and observer-dependent factors underlying intention formation. My preliminary annotation study and analyses highlight substantial variability in human interpretations and clear differences between human and model-generated narratives, underscoring the need for models that can represent ambiguity, subjectivity, and multiple plausible intentions.