Skip to content

EgoTSR++: Egocentric Spatiotemporal Reasoning for Task Progress Understanding

Sep 2026 · arXiv.org · Vol abs/2609.31751 · 0 citations · 54 references
Computer Science

TL;DR

The results establish goal-conditioned state comparison as an explicit formulation of egocentric spatiotemporal reasoning for task-progress understanding as part of a unified framework for diagnosing and improving order-robust task-progress understanding.

Abstract

Vision-Language Models (VLMs) have advanced rapidly in static visual understanding, yet remain unreliable when judging how an egocentric task is progressing. Given a task instruction and two visual observations, a model should determine which state is closer to the goal by analyzing task-relevant object configurations and spatial relations, rather than relying on timestamps or presentation order. This distinction is critical in manipulation, where retries, corrective actions, and temporary regressions make progress inherently non-monotonic. We introduce EgoTSR, a unified framework for diagnosing and improving order-robust task-progress understanding. First, SpatialLogic-Bench evaluates each physical state pair in both original and order-swapped presentations across short- and long-horizon settings, exposing whether a model follows task-state evidence or chronological shortcuts. Second, our data construction pipeline converts successful, approximately monotonic manipulation and first-person trajectories into bidirectional supervision; LongTag further preserves intermediate subtask structure for long-horizon comparison, while failure-aware data extend learning to regressions and recoveries. Third, a progressive CoT-to-Tag curriculum first supervises evidence-grounded interpretation of task-relevant state changes and then consolidates the comparison rule through scalable label-only training. Experiments reveal substantial input-order bias in representative VLMs. EgoTSR achieves 92.4% long-horizon accuracy with a 0.1-point forward-inverse Gap. Failure-aware supervision further improves accuracy on non-monotonic trajectories by 11.8 points and Recovery Accuracy by 11.2 points, while maintaining broad visual and spatial capabilities. These results establish goal-conditioned state comparison as an explicit formulation of egocentric spatiotemporal reasoning for task-progress understanding.

View source

Similar papers

Preprint Aug 2026

AtlasVLA: Persistent World-Ego State Modeling for Vision-Language-Action Models

AtlasVLA is a novel framework that transitions from direct reactive manipulation to proactive reasoning through a persistent world-ego state and decisively outperforms multi-view baselines, yielding absolute success rate improvements of 9.4% on LIBERO-Long and 17.5% in real-world long-horizon tasks.

Guiyu Zhao, Long-Teng Guo, Yang-Hong Mei et al. · 1 citation
Preprint Sep 2026

EgoPathBench: Evaluating Zero-Shot Egocentric Waypoint Decision-Making in Vision-Language Models

EgoPathBench, a dataset and five-task benchmark for first-person waypoint decision-making, and Fine-tuning Qwen 3.5 4B on the released training split improves all four reported evaluations across three external spatial benchmarks, with gains of 1.4--9.6 points.

Yang Zhao, Zhuo Chen, Xu-Bo Yang · 0 citations
Open access Aug 2026

Thinking outside the frame: viewpoint-aware spatial reasoning in large multimodal models

This work proposes a lightweight viewpoint-aware self-correction strategy with low token overhead that serves as an inductive signal, sensitizing the model to frame-of-reference changes and guiding a corresponding shift in its reasoning trajectory.

Ke Yan, Fatemeh Shiri, Zhang Hu et al. · 0 citations

Seek Before You Move: Evidence Seeking for Progress Grounding in Vision-Language Navigation

This work proposes SeekVLN, an evidence-seeking framework that couples semantic progress reasoning with active acquisition of task-relevant observations and introduces Counterfactual Contrastive Policy Optimization (C2PO) for reinforcement fine-tuning.

Zhi-Min Wang, Mei-Yuan Zhu, Duo Wu et al. · 0 citations
#machine learning Preprint Sep 2026

TEMPO: Learning Temporal Context for Dynamic Robot Manipulation

TEMPO is proposed, which augments a pretrained VLA with two temporal inputs: a motion summary extracted from a frozen video foundation model to resolve motion ambiguity and a compact proprioceptive history to resolve state aliasing.

Zhen-Yang Feng, Jimin Heo, Erik B. Sudderth et al. · 2 citations · ⚡1
Preprint Sep 2026

SpatialSkill: Self-Evolving Skills for Cross-View Spatial Reasoning

SpatialSkill is proposed, a weight-update-free framework that enables a frozen vision-language model to accumulate explicit natural-language reasoning skills from offline trajectories without modifying model parameters.

Rui-Fan Zuo, Guo-Cheng Hu, Wan-Shui Gan et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.