This work introduces SeeQ (Subtask-elicited Q-functions), which instead learns Q-values for the currently active subtask, which substantially improves best-of-N policy steering on real-world manipulation tasks.
Abstract
Despite rapid progress, generalist robot policies remain brittle on complex, long-horizon tasks that comprise multiple stages or require repeated attempts and deliberation on the same underlying stage before success. Q-value functions can improve these policies by ranking candidate actions or guiding policy improvement, but learning from sparse task-level rewards entails long credit-assignment horizons, difficult Bellman backups, and broad data-coverage requirements. We introduce SeeQ (Subtask-elicited Q-functions), which instead learns Q-values for the currently active subtask. This shortens the value-prediction horizon and enables effective learning with temporal-difference (TD) objectives. During training, subtask-level annotations present in offline robot data provide the decomposition and enable learning from broad, potentially suboptimal robot datasets. To eliminate the need for human annotations or modular subtask prediction systems at test time, our Q-function architecture is trained to autoregressively predict the active subtask in natural language before estimating its value. We instantiate SeeQ using a base vision-language backbone, pretrain it on diverse open-source robot manipulation data, and finetune it on downstream tasks. Across four real-world manipulation tasks on two bimanual robot platforms, the SeeQ value function substantially improves best-of-N policy steering.
Hierarchical planning frameworks combine skills from multiple robot control policies for long-horizon task execution, where determining when to terminate the current skill and advance to the next subtask is essential. Existing approaches often rely on pre-designed completion signal checkers that are hard to obtain in r...
Jinbang Huang, Yuan Hu, Zhi-Yuan Li et al.· arXiv.org· 1 citation
PARTS (Policy Adaptation with RL on Targeted Subtasks), a real-world subtask RL framework that concentrates practice at bottlenecks while allowing training rollouts to proceed with minimal human intervention, is presented.
Sichang Su, B. Yang, Zhi-Yun Deng et al.· 0 citations
T1, a Mixture-of-Experts model of 122B total trained with reinforcement learning, operating a real shell in a cloud sandbox for up to 300+ tool-call turns per task, rewarded by executing each task's own verifier by executing each task's own verifier.
Jun-Yao Yang, Yu-Cheng Shi, Zhong-Zhi Li et al.· 9 citations
FIERCE is presented, a generalist-initialized reinforcement learning framework centered on a unified, task-adaptive progress-failure evaluator that separates feedback quality, policy-learning efficiency, and deployment cost across simulation and two contact-rich real tasks.
Run-Jia Tan, Yuang Tu, Yu-Jie Yan et al.· arXiv.org· 0 citations
Emerging long-horizon agentic tasks require repeated model calls, worsening the inference cost of already-costly language models. While narrow agentic tasks suggest potential for aggressive model pruning without performance drop, empirical results show existing methods proposed for question answering tasks severely deg...
Yuan-Zhe Li, Peng-Xin Wang, Yuxin Ren et al.· 0 citations
Recent advances in robot learning have enabled generalist control policies capable of completing a wide range of tasks. However, their performance degrades when deployed in unseen environments, making it critical to detect failures and teach recovery behaviors. Existing runtime monitoring methods often require task- an...
Ryan Li, Yigit Korkmaz, Erdem Biyik· 2 citations· ⚡1
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.