Generic modeling of Human-Machine Interaction
TL;DR
It is suggested that making interaction structure explicit is a promising direction for multimodal interaction understanding and provides a basis for future extension toward more complex multi-party and human-robot interaction scenarios.
Abstract
Human interaction understanding is usually studied separately for Human-Human Interaction (HHI), Human-Object Interaction (HOI), and Human-Robot Interaction (HRI), which limits progress toward more general interaction models. This paper investigates whether explicit structured interaction cues can support a shared multimodal representation across interaction settings. The proposed framework encodes pose, object, relation, and temporal cues into interaction tokens and aligns them with a multimodal backbone through a lightweight trainable interface. The current study evaluates this formulation on coarse-grained dyadic interaction understanding in two complementary settings: video-based HHI on the mutual-action subset of NTU RGB+D 120 and image-based HOI on HICO-DET. Preliminary results show that the framework can generate interaction-aware outputs in both settings, with substantially stronger performance on NTU RGB+D 120 than on HICO-DET. These findings suggest that making interaction structure explicit is a promising direction for multimodal interaction understanding and provides a basis for future extension toward more complex multi-party and human-robot interaction scenarios.