Kristina Kobrock

Investigating Multimodal Informativity under Different Partner Visibility Conditions in Video-Mediated Dialogue

The results show that gesture alone is predictive of the intended referent and that multimodal fusion is most beneficial when the transcript-based model is uncertain, and that training-only alignment of learned representations with the referent image further improves the fusion model performance.