A novel unified framework, SAGE (Synchronized Action-GazE), which integrates simultaneous recognition and anticipation of both HOI and human gaze into a single unified end-to-end trainable model, laying the groundwork for more intuitive human-machine interaction.
Abstract
Human object interaction (HOI), gaze pattern, and their anticipation are intricately linked, providing valuable insights into cognitive processes, intentions, and behavior. However, most existing models handle gaze and actions separately, missing both their interdependence and the advantages of a unified solution. This paper presents a novel unified framework, SAGE (Synchronized Action-GazE), which integrates simultaneous recognition and anticipation of both HOI and human gaze into a single unified end-to-end trainable model. Our approach leverages a transformer-based architecture and incorporates gaze data into spatiotemporal attention mechanisms to simultaneously predict current and future human actions and gaze behavior. We explore this bidirectional relationship between gaze and actions under different scenarios, whether requiring a close-up, detailed view (egocentric) or a wider, more contextual view (exocentric), making our framework versatile for various applications. Additionally, due to lack of datasets for comprehensive analysis of both HOI and gaze in exocentric videos, we establish a new benchmark Exo-Cook to facilitate further research in this domain. Experiments on three benchmark datasets: VidHOI, EGTEA Gaze+, and Exo-Cook show that jointly modeling gaze and actions across current and future frames achieves consistently strong results, often surpassing specialized state-of-the-art models tailored to individual tasks. By unifying actions and attention in a comprehensive way, our work lays the groundwork for more intuitive human-machine interaction.
G3Ego, a graph-based framework for egocentric action understanding that uses gaze as a structural cue to identify action-relevant entities in the scene, achieves competitive performance compared with video-based approaches and consistently improves Macro-F1 under class-imbalanced evaluation, while avoiding reliance on computationally expensive video pretraining.
Marko Haralović, Akash Ramakrishnan, E. T. Martínez· 0 citations
A novel egocentric and exocentric video dataset capturing real-world collaboration in cooking scenarios, and establishes benchmarks for Joint Attention Estimation, Socially Conditioned Object Interaction Anticipation, and Collaborative Handover Prediction, enabling research on multimodal perception, proactive assistance, and collaborative planning.
Alexey Gavryushin, Dingxi Zhang, Zhao Huang et al.· 0 citations
This survey presents a critical review of VLMs for egocentric video understanding, tracing the progression from conventional recognition architectures to multimodal foundation models and embodied systems, and examines how first-person perception and multimodal foundation models support wearable assistance, robot skill learning, human-to-robot transfer, and embodied decision making.
Gaze target estimation, the task of predicting where a person is looking in a scene, is crucial to understanding human attention and intent. It is a challenging task that combines high-level understanding of global scene semantics and precise spatial reasoning using human appearance (e.g. pose, eye orientation). As a result, human-level performance remains elusive for existing models, limiting their practical application. To this end, we propose PaGE (Practical Gaze Estimator), a gaze estimation model that explicitly models the complex interaction between scene and head features. Using a PaGE model with a large ViT-H+ backbone as the teacher, we further distill student models with lighter backbones on a much larger and more diverse unlabeled dataset. The architectural improvements and novel training recipe allow PaGE to achieve state-of-the-art performance on several gaze estimation tasks, outperforming humans in 7 out of 9 metrics while reducing the human-AI gap by at least 60% in the remaining 2. The distilled student models retain most of the teacher's performance while being lightweight enough for practical deployment on robots and consumer devices. The code and model checkpoints are available at our project page.
Zhoutong Ye, Chengwen Zhang, Zhaibin Cui et al.· 0 citations
The increasing prevalence of wearable cameras has driven the development of egocentric (first-person) systems that assist human activities proactively by anticipating imminent interactions. A central challenge in this domain is active object interaction anticipation from first-person video-predicting what interaction will occur, when it will happen, and where it will take place. This involves forecasting (1) what interaction category (verb-noun pair), (2) when (time-to-interaction), and (3) where (the active object's location, bounding box in the last observed frame). However, existing approaches rely on limited prior knowledge about active objects and their state changes, and they struggle to (1) predict diverse state-change interactions, (2) handle temporal uncertainty of changes, and (3) localize the active object accurately in the presence of spatial distractors. To address these problems, we propose ST-KAD. It consists of a Spatial-Temporal Knowledge Aggregator that integrates rich commonsense priors to enhance what-when-where interaction anticipation and guides the model's attention toward informative cues, and a Teacher-Student Distillation framework that enables efficient inference without access to oracle inputs by transferring knowledge from an oracle-informed teacher model to a query-based student decoder. On two egocentric anticipation benchmarks (Ego4D-STA, EPIC-Kitchens-STA), ST-KAD sets a new state of the art, demonstrating accurate what-when-where prediction of future interactions. Moreover, results on four active object detection benchmarks (Ego4D-AOD, EPIC-Kitchens-AOD, MECCANO, 100DOH) further confirm that our prior-informed aggregation and teacher-student distillation generalize beyond anticipation to spatial localization, validating the generality of the design.
Yang Liu, Dejie Yang, Minghang Zheng et al.· IEEE Transactions on Pattern...· 0 citations
Hand-object interaction (HOI) recognition requires capturing both hand manipulations and object transformations. However, existing video-language models often fall into shortcuts by relying on spurious correlations among hands, objects, or environmental context, rather than reasoning from the appearance and dynamics of hands and objects themselves. To address this limitation, we propose a new learning paradigm that combines (i) hand-object masked training, which enables robust reasoning from partial hand or object observations, and (ii) an HOI-dynamics-aware decoder that explicitly learns hand- and object-centric embeddings through auxiliary predictions of their locations and semantics, enhancing sensitivity to both cues. To systematically evaluate such cue-specific reasoning, we introduce Cue-Isolated HOI (CI-HOI), a new evaluation that assesses models'ability to predict actions from hand- and object-related cues independently. To enable CI-HOI, we curate the DEHOI testbed, which separates hand- and object-related observations for disentangled HOI evaluation through inpainting. Using DEHOI, we demonstrate both quantitatively and qualitatively that our training strategy exploits hand- and object-centric information more effectively than existing models. Our approach improves over existing models on DEHOI, standard action recognition, object state recognition, and even robot manipulation action recognition, leading to more robust HOI understanding.
Masatoshi Tateno, Alexandros Stergiou, Risa Shinoda et al.· 0 citations