NVFCorpus: A Corpus for Analyzing Multimodal and Multifunctional Nonverbal Behavior in Multiparty Conversations

Abstract

This paper presents the Nonverbal Function Corpus (NVFCorpus), a multimodal conversation corpus for exploring the multifunctional aspects of nonverbal behaviors. This corpus is based on a novel methodological paradigm, nonverbal functional spectrum analysis (NFSA). NFSA examines two intrinsic properties of nonverbal behaviors: functional multiplicity, in which a single behavior can simultaneously serve multiple functions, and interpretive diversity, in which the same behavior can be interpreted differently by different observers. To address these properties, the concept of the functional spectrum is introduced as a third-party interpretable functional distribution over multimodal, multifunctional nonverbal behaviors in conversations. The NVFCorpus focuses on four-person consensus-building dialogues among six groups of mixed-gender Japanese participants, each of whom participates in four 10-minute sessions conducted in both face-to-face and remote settings. In the face-to-face sessions (125 minutes in total), three external raters assigned nonexclusive functional labels covering more than 120 functions across head movements, facial expressions, gaze, and listener backchannels (Aizuchi) at a fine temporal resolution. The resulting functional interactions, represented as functional spectra, serve as an intermediate representation that connects observable physical behaviors with social outcomes within the hierarchical model of dialogic meaning creation proposed in this study. This hierarchical structure introduces several new research challenges, such as the automatic inference of consensus-building dialogue quality from observable behaviors. NVFCorpus and its functional-spectrum paradigm offer a new public resource for investigating the socially interpretable roles of multimodal nonverbal behaviors.