Skip to content

Hide-and-Seek in Trajectories: Discovering Failure Signals for VLA Runtime Monitoring

May 2026 · arXiv.org · Vol abs/2605.30834 · 8 citations · 95 references
Computer Science

TL;DR

This paper proposes Hide-and-Seek, a framework that formulates VLA failure detection as a coarsely supervised learning problem that achieves state-of-the-art multi-task failure detection performance with a practical accuracy--timeliness trade-off under conformal prediction, and generalizes well to both seen and unseen tasks.

Abstract

Vision-Language-Action (VLA) models enable robots to follow natural language instructions and generalize across diverse tasks, but they remain vulnerable to execution failures that compromise reliability in real-world deployment. Detecting such failures during execution is therefore critical for the robust deployment of embodied systems. Existing failure detection methods either rely on expensive action resampling or external models, while alternatives propagate trajectory-level labels uniformly across every timestep, obscuring localized failure signals. In this paper, we propose \textbf{Hide-and-Seek}, a framework that formulates VLA failure detection as a coarsely supervised learning problem. By combining inter-trajectory and intra-trajectory contrastive objectives, Hide-and-Seek localizes failure-indicative actions and induces temporally structured failure signals from trajectory-level supervision alone, without any step-level annotation. We evaluate Hide-and-Seek on LIBERO, VLABench, and a real-world robotic platform across three representative VLA policies: OpenVLA, $\pi_0$, and $\pi_{0.5}$.Our method achieves state-of-the-art multi-task failure detection performance with a practical accuracy--timeliness trade-off under conformal prediction, and generalizes well to both seen and unseen tasks.

View source

Similar papers

Preprint Sep 2026

FailureSpot: Label-Efficient Timestamp-Level Failure Detection for Vision-Language-Action Models

Vision-language-action (VLA) policies have shown strong potential for general-purpose robotic manipulation, but they can still fail unpredictably during long-horizon execution, making reliable failure detection essential for safe deployment. Existing methods either rely on visual models that typically detect failures o...

Jie Ma, Zong-Xi Liu, Yi Zhu · 0 citations
Sep 2026

ProAct-VLM: Pre-Failure Vision-Language Task Replanning with Continuous Perception Feedback

Long-horizon robotic tasks are vulnerable to unexpected environmental changes that can render planned actions ineffective or unsafe. To address this, robots must detect such changes as they occur, interpret their impact, and adjust their actions accordingly. Traditional rule-based decision-making pipelines are brittle...

A. N. Ahmed, Omar Moured, Mughni Irfan Mohammed Abdul et al. · 0 citations
Preprint Aug 2026

Drive the Thoughts: Runtime Monitoring of VLA Reasoning-Trajectory Consistency

Autonomous vehicles (AVs) operate in complex environments where failures are consequential. Sophisticated machine learning models for perception and planning are key to overcoming at least part of that complexity, but their black-box nature complicates validation and verification (V&V). The recent integration of Vision...

Tianwen Yu, Lu Feng, Sebastian G. Elbaum · 1 citation
Preprint Sep 2026

VLA-Corrector: Stage-Aware Observable State Understanding for Prompt-Based Closed-Loop Recovery of Vision-Language-Action Policies

Results show that observable visual-proprioceptive-action history is sufficient to infer latent task states and enable practical failure recovery for existing VLA policies, and the proposed approach substantially improves closed-loop reliability under diverse perturbations.

Chang-Nan Song, Bin Qian, Yan Feng et al. · 0 citations
Sep 2026

SEES: A Self-Evolving Embodied System via Failure-Guided VLA Policy Adaptation

A Self-Evolving Embodied System (SEES) that learns from failures and improves the VLA policy without additional expert demonstrations, and can be integrated with different VLA backbones to progressively improve their long-horizon performance.

Zi-Wen Li, Hanlue Zhang, Zhen-Yang Ren et al. · 1 citation

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.