Skip to content

Video2Skill: From Streaming Experience to Reusable Embodied Skills

Sep 2026 · 0 citations · 70 references
Computer Science

TL;DR

This work introduces Video2Skill, a benchmark that covers robot tabletop manipulation and human kitchen activity and tests three core capabilities: locating manipulation events in time, grouping events of the same transformation, and deciding when to reuse an existing skill or create a new one.

Abstract

Manipulation behaviors vary widely across objects and scenes, but they share a small set of reusable skills, and planning with these skills helps embodied agents generalize to new tasks. Yet an agent can only plan with skills it knows. Recovering skills from observed experience, the inverse of planning, builds this knowledge over time and yields skill data for training future agents. Vision-Language Models (VLMs) describe individual manipulation events well, but can they organize a stream of events into reusable skills? We formulate this problem as Streaming Embodied Skill Discovery (SESD): a model watches videos in sequence and maintains a persistent skill library that shapes its later decisions. To systematically measure this ability, we introduce Video2Skill, a benchmark that covers robot tabletop manipulation and human kitchen activity and tests three core capabilities: (i) locating manipulation events in time, (ii) grouping events of the same transformation, and (iii) deciding when to reuse an existing skill or create a new one. Across 19 open-source VLMs, many models group events at near-chance level, and scale does not consistently help. Their errors depend on how perception and library updates are coupled: joint models merge distinct transformations into one skill, while models that update the library from text descriptions duplicate recurring ones. Supervised fine-tuning, including our counterfactual library-state rebalancing (CLaRe), improves grouping but exposes a deeper bottleneck: trained models consolidate familiar skills yet rarely expand the library. Their libraries stall below half the reference size, and transformations unseen in training are located in time but almost never given a new skill. Recognizing when existing skills are insufficient thus emerges as the central challenge.

View source

Similar papers

Sep 2026

CHOREO: Every Humanoid Skill as a Trajectory

CHOREO, a framework for training-free composition of heterogeneous humanoid skills, demonstrates that executable trajectories provide a scalable interface for accumulating and composing pretrained humanoid capabilities.

Zi-Yi Sun, Jing-Wen Chen, Yu-Xin Wang et al. · 1 citation · ⚡1
#artificial intelligence Preprint Oct 2026

Recursive Video In-Context Learning for Agentic Robot

LLM agents that orchestrate frozen vision-language-action (VLA) policies improve across episodes through text memory, which records what the agent did but not how the task is done. A demonstration video shows it, but fits poorly into an agent's context. The full video slows every turn, fixed keyframes lose the contact...

Wen-Rui Bao, Xin-Xin Liu, Bing-Xin Xu et al. · 1 citation
Preprint Sep 2026

MessyMem: Learning-from-Doing Memory for Mobile Manipulation

Mobile manipulators deployed across many rooms and visits should improve with experience: after discovering that a cabinet is locked or finding an object in a drawer, the robot should reuse that knowledge rather than start each task from scratch. Yet today's robots often treat each task as new: compact scene representa...

Anuva Banwasi, William Muckelroy, Priya Sundaresan et al. · 0 citations
#machine learning Preprint Aug 2026

PRACTICE: From Experience to Expertise in Self-Evolving Embodied Agents

This work introduces PRACTICE, which trains a skill learner to discover and maintain a persistent skill library from past interaction trajectories while keeping the task executor frozen, and demonstrates that a compact skill learner delivers consistent performance improvements across successive library-update rounds fo...

Ziyi Bai, Si-Qi Li, Ting-Lei Huang et al. · 0 citations

Video2STL: Grounding VLM-Generated Temporal Specifications for Robot Learning

Video2STL is presented, a framework that converts observation-only videos into parametric Signal Temporal Logic (STL) specifications and uses the resulting formal representation for robot learning, and supports cross-embodiment transfer from human or animal videos to robot control.

Merve Atasever, Keyan Azbijari, Cagan Bakirci et al. · 0 citations
Preprint Sep 2026

Uruqi: Learning Spatial Cognition from Visual Experience

Spatial intelligence requires maintaining a coherent understanding of the world as the embodied agent moves. Like humans, the agent must use its own motion to interpret changes across observations and update object locations and spatial relations accordingly. Despite spatial post-training having substantially broadened...

Shi-Chao Li, Mei-Qi Wang, Fei Su et al. · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.