A framework for learning human-like robot motion from demonstration, including data collection, probabilistic trajectory learning, and perceptual user evaluation is presented, extending the widely used Gaussian Mixture Model and Gaussian Mixture Regression approach for learning from demonstration.
Abstract
Learning from demonstration (LfD) provides a developmental framework through which robots can develop motor skills by observing and imitating human dynamics, reducing reliance on explicit programming to teach a skill to a robot. The resulting human-like robot motion is recognised as a key factor in building trust and enabling natural collaboration in human-robot interaction. This paper presents a framework for learning human-like robot motion from demonstration, including data collection, probabilistic trajectory learning, and perceptual user evaluation. A dataset of 3,142 handwriting demonstrations was collected from 22 participants across all 52 Latin alphabet character-case combinations via a touchscreen teleoperation interface, capturing planar position, contact force, and timing. Building on the widely used Gaussian Mixture Model and Gaussian Mixture Regression approach for learning from demonstration, the framework is extended in this work by incorporating force and normalised time dimensions to enable richer representation of human dynamics, and adapting it to handle non-continuous, multi-segment trajectories, enabling generalisation across demonstrations. A user study with 21 participants evaluated the perceived human-likeness of the generated trajectories using a continuous scale anchored between robotic and human-like motion, normalised to 0-100 where 50 represents the neutral midpoint. The generated trajectories achieved an overall human-likeness score of 71.50 (SD=22.56), indicating that the majority of trajectories were perceived as more human-like. Participants identified geometric positioning and trajectory sequence as the most influential perceptual factors, and reported positive attitudes toward human-like robot behaviour. The datasets are released as open-source, providing a reproducible benchmark for developing and evaluating human-like robot motion methods.
Learning from Demonstrations (LfD) enables robots to acquire new skills by imitating a domain expert, most commonly a human. In practice, however, human demonstrations are typically available only under a limited budget and rarely cover the full range of situations a robot may encounter. This limitation often induces a distribution shift between the states represented in offline demonstrations and those visited by the robot during online exploration, which can substantially degrade performance. Interactive Imitation Learning (IIL) mitigates this shift by keeping the human teachers in the learning loop, allowing them to provide online input such as feedback and corrections. Yet these benefits come at the cost of sustained human supervision and can be undermined by noise, inconsistency, and errors in human teaching.
This thesis investigates Active Learning from Demonstrations (Active LfD) as a principled approach to reduce distribution shift while accounting for human factors in realistic human-in-the-loop settings. In Active LfD, the robot learner does not passively consume demonstrations. Instead, it optimizes its query decisions to selectively request demonstrations from a human teacher. By determining when and what to query, the learner aims to reduce reliance on continuous monitoring, focus human effort where it is most beneficial, and limit the impact of suboptimal human teaching decisions.
The thesis is organized into three parts. In Part I, I lay the foundation for how a robot learner can actively shape human teaching by revisiting conventional offline LfD through the lens of demonstration distributions. After identifying biased human teaching strategies under unguided conditions, I develop an Active LfD algorithm that operates with offline demonstrations and guides the resulting demonstration distribution toward a more balanced one, or more generally toward any specified target distribution, so as to improve robot learning. Importantly, however, a ``balanced'' distribution (e.g., a uniform distribution used as a default) is not necessarily optimal for learning. The most beneficial demonstration distribution depends on the robot’s evolving policy and its online exploration, and is therefore difficult to specify a priori. To address this coupling, I present a second Active LfD algorithm that leverages online demonstrations, iteratively deciding both the timing and content of queries as learning progresses in order to optimize the demonstration distribution for the learner.
In Part II, I explore how Active LfD extends to more realistic and complex scenarios of learning from human teachers. I first extend the paradigm to transfer learning for complex manipulation tasks, spanning a range of policy transfer scenarios. I then consider the practical case of non-expert human teachers, who may provide imperfect demonstrations relative to the task objective. To accommodate imperfect teaching, I develop an Active LfD framework that optimizes the query sequence by jointly accounting for both the expected improvement in the robot’s policy induced by a query and the teacher’s capability to provide informative guidance in the queried region.
Part III closes the loop of human-robot interactive learning by examining how robot query design influences human teaching beyond user experience alone. I leverage Curriculum Learning to design an Active LfD algorithm that benefits both robot learning and human teaching, encouraging a reciprocal loop between the robot learner and the human teacher.
Large-scale pre-training has made robot policy fine-tuning increasingly data-efficient, but this progress has largely been driven by datasets and embodiments built around simple parallel-jaw grippers. Dexterous, multi-fingered hands remain comparatively data-starved because real teleoperation is costly to scale, while human hand video is off-embodiment and requires lossy pose estimation and retargeting. We introduce Simulation Pre-training for Dexterity (SPD), a pre-training framework for dexterous manipulation that uses data entirely collected in simulation. In SPD, humans manipulate virtual objects inside a VR headset, enabling on-embodiment trajectories and robot-free collection. With the help of five operators, we collect 75 hours of multi-task dexterous manipulation over one week, and use it to pre-train a causal transformer on a sequence modeling objective. We study the benefits of simulation pre-training on real-world tasks by fine-tuning on 1-2 hours of physical demonstrations on a 56-DoF bimanual dexterous setup. We find that our approach outperforms training behavior cloning policies from scratch, showing that simulation teleoperation is a viable pre-training source for real-world dexterous manipulation. We perform ablation studies, measuring the benefits of history conditioning and short action chunks for reactive control.
Sarthak Kamat, Adam Rashid, Satvik Sharma et al.· 0 citations
The ability to acquire skills rapidly and effortlessly while retaining those already mastered is essential for robots. However, current methods still rely on a cumbersome training-time loop that is costly and slow, while eroding skills already mastered. In this paper, we introduce HOST (Human-to-robot One-Shot Skill AcquisiTion), a framework that enables a robot to acquire skills in seconds from a single human video while retaining previously mastered skills. HOST resolves skill acquisition through a cascade of self-grounded prediction. It first estimates the robot's progress within the demonstrated task, then translates the upcoming progression into the robot's own future observations, and finally derives actions from these predicted observations. This cascade is trained on targets coupled to the video demonstration, obtained by mapping the robot trajectory and the video demonstration onto a shared task progress manifold, then redefining each target to align with the future progression of the video. HOST thereby enables the robot to actively follow the demonstrated procedure and adapt it to the robot's embodiment. HOST acquires novel skills at inference time from a single human video in an average of 29 seconds and achieves a 62% average success rate. It exceeds the zero-shot baseline by 45% while retaining previously mastered skills. HOST even exceeds the baseline fine-tuned on 50 robot demonstrations per task while requiring 50 times fewer demonstrations and acquiring each skill 507 times faster. Additional information about HOST is available on the project website.
Guangyan Chen, Meiling Wang, Te Cui et al.· 1 citation
Bimanual teleoperation provides an effective way to collect robot demonstrations, but it assumes that the operator and robot have matching numbers of simultaneous control channels. This assumption breaks for tri-manual systems: the robot can coordinate three arms concurrently, whereas a single operator can continuously control only two. Pairwise mode switching may therefore record otherwise independent motions sequentially, causing behaviour cloning to reproduce delays imposed by the interface rather than required by the task. We present TriManPolicy, a tri-manual imitation learning system that allows one operator to demonstrate behaviours for three arms. Its central component is Dependency-Aware Tri-Arm Scheduling (DATS). The key idea is to preserve the demonstrated arm motions while reconsidering when they occur. DATS retimes demonstrations offline by preserving local sensorimotor segments of fixed duration and repositioning them according to constraints on task order and arm usage that are reviewed by a human. The resulting data train a single synchronous policy for all three arms, while deployment requires neither the dependency graph nor the scheduler. Across six challenging tasks performed in the real world, policies trained on demonstrations retimed by DATS exhibit more efficient coordination while maintaining comparable observed task success. Offline analysis further shows that DATS changes the supervision across arms rather than merely removing idle periods. Project videos and additional material are available at https://aus.bot/trimanpolicy/.
Learning from Demonstration (LfD) allows robots to learn manipulation tasks directly from humans, thereby supporting the versatile application of robots. Most LfD methods do not explicitly model the physical interactions between a robot and its environment, such as the making and breaking of contact, while these are crucial during manipulation tasks. Because the same basic physical interactions recur often, they can be a basis for robust, generalizable, and adaptive task reproduction. We propose an LfD method that explicitly uses what physical interactions take place where and when. Using that information, a hybrid position-force controller tracks demonstrated trajectories until contact-based transition conditions from the demonstrations are met. We evaluate our method in real robot experiments consisting of opening doors and locks, bolt picking and screwing, dislodging, and surface contouring. We show that explicitly modeling physical interactions benefits LfD in four ways. First, by allowing reproduction of complex, sequential, and contact-rich manipulation tasks using only a single demonstration and no prior knowledge of the task. Second, by facilitating robustness to unknown geometric variations in the environment. Third, by facilitating generalization when geometric variations are known. Fourth, by facilitating online adaptation using geometric information explored during task reproduction. We discuss how robustness, generalization, and adaptivity can be explicitly implemented, which is generally lacking in the LfD literature. Thereby, our work aims to close a gap in interpretable few-shot LfD of robotic manipulation.
A. H. G. Overbeek, H. V. D. Kooij, M. Vlutters· 0 citations
This work introduces a novel multi-modal orchestration framework for semantic audio-driven humanoid control, enabling robots to autonomously select and execute appropriate motion skills in real time.
J. Marcelo, M. Brienza, E. Bugli et al.· 0 citations
