Dyad-Specific Multimodal Affective Dynamics: Integrating Temporal Cross-Modal Consistency and Probabilistic Agreement

Abstract

Repeated dyadic interactions are characterized not only by average multimodal signal levels but also by how cross-modal consistency evolves over time. This study investigated whether the joint temporal organization of within-person face–body cross-modal consistency and probabilistic cross-modal affective agreement, quantified by the Consistency Index for Affective Synchrony (CIAS) and the Probabilistic Multimodal Consistency Index (PMCI), yielded reproducible session-level dynamic representations associated with repeated dyadic interaction. The primary analysis included 24 improvised and 17 naturalistic sessions, with 12 dyads in each condition. Time-varying signals from both partners were summarized through dynamic-state occupancy, dwell structure, transition characteristics, and temporal variability and evaluated using relational permutation analysis with shared-participant and technical controls. In naturalistic interaction, repeated sessions of the same dyad were significantly more similar (β = −2.0180, FDR q = 0.0114), with significant dyad-specificity beyond shared-participant similarity (Δβ = −2.4670, q = 0.0033). Representation ablations showed that PMCI provided the dominant state-separation structure, whereas CIAS contributed modest and condition-dependent complementary information. Most notably, frozen participant-disjoint evaluation on 73 naturalistic sessions from six dyads and 11 previously unseen participants retained significant dyad-associated structure and achieved session-weighted nearest-dyad accuracy of 0.8356 and macro-dyad accuracy of 0.8512. Additional robustness analyses showed that the principal pattern persisted across state-count specifications, temporal-window settings, partner-order reversal, and explicit technical-quality adjustment. Overall, the findings support the existence of reproducible session-level multimodal dynamic structure associated with repeated dyadic interaction and its transfer to previously unseen participants, while not implying direct interpersonal synchrony or psychologically defined interaction states.