Active Robot Learning from Demonstrations with Human Teachers

Abstract

Learning from Demonstrations (LfD) enables robots to acquire new skills by imitating a domain expert, most commonly a human. In practice, however, human demonstrations are typically available only under a limited budget and rarely cover the full range of situations a robot may encounter. This limitation often induces a distribution shift between the states represented in offline demonstrations and those visited by the robot during online exploration, which can substantially degrade performance. Interactive Imitation Learning (IIL) mitigates this shift by keeping the human teachers in the learning loop, allowing them to provide online input such as feedback and corrections. Yet these benefits come at the cost of sustained human supervision and can be undermined by noise, inconsistency, and errors in human teaching. This thesis investigates Active Learning from Demonstrations (Active LfD) as a principled approach to reduce distribution shift while accounting for human factors in realistic human-in-the-loop settings. In Active LfD, the robot learner does not passively consume demonstrations. Instead, it optimizes its query decisions to selectively request demonstrations from a human teacher. By determining when and what to query, the learner aims to reduce reliance on continuous monitoring, focus human effort where it is most beneficial, and limit the impact of suboptimal human teaching decisions. The thesis is organized into three parts. In Part I, I lay the foundation for how a robot learner can actively shape human teaching by revisiting conventional offline LfD through the lens of demonstration distributions. After identifying biased human teaching strategies under unguided conditions, I develop an Active LfD algorithm that operates with offline demonstrations and guides the resulting demonstration distribution toward a more balanced one, or more generally toward any specified target distribution, so as to improve robot learning. Importantly, however, a ``balanced'' distribution (e.g., a uniform distribution used as a default) is not necessarily optimal for learning. The most beneficial demonstration distribution depends on the robot’s evolving policy and its online exploration, and is therefore difficult to specify a priori. To address this coupling, I present a second Active LfD algorithm that leverages online demonstrations, iteratively deciding both the timing and content of queries as learning progresses in order to optimize the demonstration distribution for the learner. In Part II, I explore how Active LfD extends to more realistic and complex scenarios of learning from human teachers. I first extend the paradigm to transfer learning for complex manipulation tasks, spanning a range of policy transfer scenarios. I then consider the practical case of non-expert human teachers, who may provide imperfect demonstrations relative to the task objective. To accommodate imperfect teaching, I develop an Active LfD framework that optimizes the query sequence by jointly accounting for both the expected improvement in the robot’s policy induced by a query and the teacher’s capability to provide informative guidance in the queried region. Part III closes the loop of human-robot interactive learning by examining how robot query design influences human teaching beyond user experience alone. I leverage Curriculum Learning to design an Active LfD algorithm that benefits both robot learning and human teaching, encouraging a reciprocal loop between the robot learner and the human teacher.