Robust Instant Policy: Student’s-t-Based Trajectory Aggregation for Reliable In-Context Imitation Learning in Robot Manipulation

Abstract

Imitation learning (IL) enables robots to acquire autonomous skills from human demonstrations. Recent in-context IL (ICIL) methods leverage off-the-shelf large language models (LLMs) as instant policies that infer task context and generate executable trajectories from only a few demonstrations, without retraining model parameters. However, the reliability of LLM-based instant policies is limited by hallucinations, which produce outlier trajectories inconsistent with the demonstrations. This issue is further amplified in multimodal tasks, where multiple distinct yet equally valid strategies exist in human demonstrations. To address these challenges, we propose the Robust Instant Policy (RIP) framework, which improves reliability by robustly aggregating multiple trajectories sampled from an LLM. As a concrete instantiation, we introduce Student’s $t$ mixture model RIP (SMM-RIP), which captures trajectory multimodality via an SMM and extracts a robust representative trajectory using Student’s $t$ regression, effectively suppressing hallucinated outliers while preserving valid strategies. Experiments on 15 everyday manipulation tasks in both simulated and real-world environments show that SMM-RIP consistently outperforms state-of-the-art IL methods, particularly in low-data regimes. These results highlight the importance of robustness and flexible multimodal modeling for reliable LLM-based ICIL. Video results are available at https://hanbit-o.github.io/research/rip.html