Skip to content
Conference Open access

EgoTools: Towards Tool-Centric Reasoning in Real-World Egocentric Videos

Sep 2026 · Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing · 0 citations · 48 references
Computer Science

TL;DR

EgoTools is introduced, the first comprehensive suite for egocentric tool-use understanding, and results establish EgoTools as a unified resource for both training and diagnostic evaluation of real-world egocentric tool-use understanding.

Abstract

Real-world embodied tasks, from everyday activities to professional procedures, require agents to act under physical constraints while tracking evolving object and task states. Tool use sits at the heart of such tasks, as many everyday and professional activities are tool-mediated. Understanding them requires reasoning about affordances, hand-tool-object geometry, procedural progress, and causal effects on target objects. Yet despite strong performance on perception-oriented video tasks such as captioning and general video QA, current multimodal video models remain limited in this form of tool-centric embodied reasoning. Progress in this direction has been limited by the lack of real-world egocentric data and diagnostic benchmarks. To address this gap, we introduce EgoTools, the first comprehensive suite for egocentric tool-use understanding. It consists of two complementary components: EgoTools-Data, a large-scale corpus of 100 hours of tool-centric egocentric recordings with synchronized audio, dense captions, reasoning-heavy narrations, and supplementary 3D information; and EgoTools-Bench, a diagnostic benchmark of 1,000 QA pairs across four tracks that cover tool-use understanding from perception and geometry to procedure and causal reasoning. Experimental results show that current models still struggle to ground tool use in visual evidence: Gemini-3.1-Pro achieves 66.9% overall accuracy but only 51.7% on Perception&Grounding. Beyond evaluation, we validate EgoTools-Data as a training resource. On the full 1,000-question benchmark, full supervised fine-tuning improves Qwen3-VL-8B-Instruct from 50.0% to 60.9%, under strict source-video separation. Together, these results establish EgoTools as a unified resource for both training and diagnostic evaluation of real-world egocentric tool-use understanding.

Read PDF

Similar papers

Preprint Oct 2026

Ego2Act: Evaluating Goal-Directed Manipulation in Egocentric Video Generation

Video generation models are increasingly being explored as world simulators for embodied planning and learning. To do so effectively, these models must not only generate visually appealing frames, but also predict how environments dynamically evolve when executing goal-directed actions. While evaluating these capabilit...

Patrick Amadeus Irawan, Iskandar Muda Rizky Parlambang, Rava Maulana et al. · 0 citations
Preprint Oct 2026

HumanPhys: A Human-Centric Physical-World Perception Benchmark for Real-World Visual Reasoning

We introduce HumanPhys, a human-centric benchmark for real-world physical perception, and a training corpus emitted by the same pipeline. Each question targets a physically grounded quantity related to people, including depth, inter-person distance, gaze angle, displacement, velocity, heading-step trajectory, blink cou...

Yi-Ming Zhang, En-Zhi Zhou, Ju-Nan Lin et al. · 0 citations
#artificial intelligence Preprint Oct 2026

EgoOmniWorld: Unifying Egocentric Understanding via Ego-Action-World Interactions

Egocentric video understanding spans diverse capabilities, modalities, and evaluation formats, yet these are often studied in isolation despite their tight coupling in real-world interaction. We propose the Ego-Action-World interaction loop as a unifying principle: the wearer perceives the world, forms goals, acts, cha...

Song-Ruo Chen, Yu-Qian Fu, Yu Li et al. · 0 citations
Preprint Oct 2026

Ego-CEVR: Learning Cross-Embodiment Visual Representations from Human Egocentric Videos

As egocentric video becomes increasingly available for embodied learning, an important research question is how to turn this low-cost, scalable source of data into robot manipulation capabilities. Egocentric videos naturally provide rich context for high-level task understanding, while real-robot demonstrations with wr...

Ke-Ru Zhou, Zhi-Rui Fang, Xin-Run Xu et al. · 0 citations
Preprint Oct 2026

EgoContact: Evaluating Physical Intelligence with Egocentric Video

This work presents EgoContact, a large-scale egocentric dataset for evaluating the generalizability and precision of learned manipulation priors through future contact-point prediction, and demonstrates the usefulness of EgoContact's annotated interactions for robotic manipulation.

Adam Deryło, Alexey Gavryushin, Daniel Cremers et al. · 0 citations
Sep 2026

EgoTSR++: Egocentric Spatiotemporal Reasoning for Task Progress Understanding

The results establish goal-conditioned state comparison as an explicit formulation of egocentric spatiotemporal reasoning for task-progress understanding as part of a unified framework for diagnosing and improving order-robust task-progress understanding.

Xiao-Da Yang, Can Wang, Yu-Xiang Liu et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.