G3Ego, a graph-based framework for egocentric action understanding that uses gaze as a structural cue to identify action-relevant entities in the scene, achieves competitive performance compared with video-based approaches and consistently improves Macro-F1 under class-imbalanced evaluation, while avoiding reliance on computationally expensive video pretraining.
Abstract
Egocentric action understanding is often addressed using large video models pretrained on extensive exocentric datasets. However, many first-person actions depend on a small number of hand-object interactions involving only a few relevant entities. We propose G3Ego, a graph-based framework for egocentric action understanding that uses gaze as a structural cue to identify action-relevant entities in the scene. From sparsely sampled frames, G3Ego constructs action scene graphs from vision-language descriptions, grounded objects, and hand cues, and then prunes irrelevant entities using the camera wearer's gaze. The resulting graph embeddings are temporally aggregated for action recognition and anticipation. Unlike prior work that uses gaze primarily as an auxiliary modality or attention signal, G3Ego incorporates gaze directly into graph construction, producing efficient and interpretable representations focused on action-relevant interactions. Experiments on EGTEA Gaze+ and MECCANO show that G3Ego achieves competitive performance compared with video-based approaches and consistently improves Macro-F1 under class-imbalanced evaluation, while avoiding reliance on computationally expensive video pretraining. These results demonstrate the effectiveness of gaze-guided graph representations for egocentric action understanding.
This survey presents a critical review of VLMs for egocentric video understanding, tracing the progression from conventional recognition architectures to multimodal foundation models and embodied systems, and examines how first-person perception and multimodal foundation models support wearable assistance, robot skill learning, human-to-robot transfer, and embodied decision making.
A novel unified framework, SAGE (Synchronized Action-GazE), which integrates simultaneous recognition and anticipation of both HOI and human gaze into a single unified end-to-end trainable model, laying the groundwork for more intuitive human-machine interaction.
TokAG, a zero-shot affordance grounding framework that exploits the token-level semantic-spatial signals in LVLMs to localize action-relevant regions without external supervision, and introduces a spatial-aware token-selection mechanism to systematically evaluate each output token.
Seung Il Lee, Qinqian Lei, Daguang Xu et al.· 0 citations
Experiments on egocentric video benchmarks show LogFA significantly improves model generalization to unseen environments while maintaining low computational and data collection costs.
Gaze target detection requires understanding where a person is looking by jointly reasoning about the gazer and the surrounding scene. While recent methods have benefited from powerful pretrained visual backbones, they often treat gaze prediction as a generic localization problem and overlook a key property of the task: the target should be interpreted in relation to the person’s head. This limits their ability to model direction, distance, and head-scene dependencies in a unified manner. We propose GazeHRNet, a head-centric reasoning framework for RGB-based gaze target detection. Instead of relying on absolute image coordinates or auxiliary geometric inputs, GazeHRNet represents the scene from the gazer’s perspective through Head-Centric Polar Encoding and organizes visual features by their spatial relevance to the head via Head-Aware Attention Routing. It further combines coarse spatial reasoning with fine-grained anisotropic heatmap prediction, enabling reliable target localization under cluttered scenes and varying head positions. Experiments on GazeFollow and VideoAttentionTarget show that GazeHRNet achieves 0.952 and 0.929 AUC with L2 distances of 0.102 and 0.103, respectively, using only RGB input and 3 M trainable parameters. Cross-dataset evaluation further demonstrates improved robustness and generalization across different scenes and subject distributions.
Tianxiang Nan, Chenglizhao Chen, Xi Chen et al.· Italian National Conference...· 0 citations
Hand-object interaction (HOI) recognition requires capturing both hand manipulations and object transformations. However, existing video-language models often fall into shortcuts by relying on spurious correlations among hands, objects, or environmental context, rather than reasoning from the appearance and dynamics of hands and objects themselves. To address this limitation, we propose a new learning paradigm that combines (i) hand-object masked training, which enables robust reasoning from partial hand or object observations, and (ii) an HOI-dynamics-aware decoder that explicitly learns hand- and object-centric embeddings through auxiliary predictions of their locations and semantics, enhancing sensitivity to both cues. To systematically evaluate such cue-specific reasoning, we introduce Cue-Isolated HOI (CI-HOI), a new evaluation that assesses models'ability to predict actions from hand- and object-related cues independently. To enable CI-HOI, we curate the DEHOI testbed, which separates hand- and object-related observations for disentangled HOI evaluation through inpainting. Using DEHOI, we demonstrate both quantitatively and qualitatively that our training strategy exploits hand- and object-centric information more effectively than existing models. Our approach improves over existing models on DEHOI, standard action recognition, object state recognition, and even robot manipulation action recognition, leading to more robust HOI understanding.
Masatoshi Tateno, Alexandros Stergiou, Risa Shinoda et al.· 0 citations