Jul 2026· Proceedings of the Special Interest Group on Computer Graphics and Interactive Techniques Conference Courses· 0 citations· 8 references
TL;DR
This short course presents a unified pipeline for developing humanoid and general-purpose robot policies, spanning synthetic data generation, policy training, and deployment, and gains a practical understanding of how simulation, world models, and foundation models compose into a scalable, end-to-end system for generalizable physical AI.
Abstract
Building intelligent robots that map sensor inputs and language instructions directly to actions, is a rapidly advancing area, but progress is bottlenecked by the cost of real-world data and the risk of physical testing. This short course presents a unified pipeline for developing humanoid and general-purpose robot policies, spanning synthetic data generation, policy training, and deployment. At the core of the pipeline, world foundation models serve a dual role: generating diverse training distributions from limited teleoperation data through controllable augmentation, and acting directly as action policies by predicting future states conditioned on language and sensory context. These capabilities feed into reinforcement and imitation learning in high-fidelity simulation, where cross-embodiment vision-language-action (VLA) models learn robust sensorimotor policies. The resulting policies are then deployed on humanoid platforms via real-time edge inference, closing the loop from data generation to physical execution. Participants will gain a practical understanding of how simulation, world models, and foundation models compose into a scalable, end-to-end system for generalizable physical AI.
End-to-end vision-language-action (VLA) and world-action models offer an elegant route to general-purpose robotics, but their reliability is bounded by validated physical coverage. When an unfamiliar object, sensor, embodiment, or contact falls outside that coverage and no validated fallback exists, correcting the failure requires new robot data, a policy update, and regression testing. This recurring burden is the retraining tax. Unlike text, embodied data must often be created by operating machines. We present Teach-and-Grow Learning (TGL), an agent-centered architecture for general robot learning. In its general form, a multimodal agent turns a few successful demonstrations into reusable Skill Blocks: closed-loop behaviors for meaningful subgoals. In a new scene, the agent grounds and composes these blocks, selects learned or geometric tools, observes the physical outcome, and revises the route when execution departs from intent. A Skill Library stores executable behavior, while structured Experience Memory carries forward success, failure, and repair. New tasks are acquired without task-specific policy retraining. Our LIBERO evaluation attains state-of-the-art performance; controlled studies expose skill induction, persistent reuse, and agent-directed adaptation. Finally, we propose the Teach-and-Grow scaling-law hypothesis: if X denotes effective reusable experience, future-task error and teaching demand should approach irreducible floors as power laws in X. The architecture therefore treats deployment as a period of continued learning, in which one task can make the next easier.
BWM is an action-conditioned world model that combines initial-environment guidance, dynamic visual history, and temporally aligned robot-action conditioning for stateful autoregressive prediction of future observations and is released as an open-source, low-cost, high-fidelity world simulator for robot manipulation.
: The integration of Deep Learning, Deep Reinforcement Learning, and massive Vision-Language-Action (VLA) foundation models has catalysed a profound paradigm shift in robotics, transitioning systems from rigid automation to dynamic, open-world autonomy. Despite transformative breakthroughs in fields such as healthcare, ranging from adaptive robotic rehabilitation to autonomous surgical manipulation and silver care, widespread real-world deployment remains severely bottlenecked. This limitation primarily stems from the “Reality Gap” inherent to sim-to-real transfer and a fundamental epistemological tension: the stochastic, “black-box” nature of unconstrained neural networks fundamentally conflicts with the deterministic, zero-violation safety guarantees demanded by physical robotics. To address these critical barriers, this comprehensive review systematically synthesises state-of-the-art algorithmic building blocks across perception, dynamics modelling, and control. Moving beyond traditional incremental surveys, we introduce unifying conceptual frameworks, such as Certified-Semantic Embodiment (CSE) and Semantic-Kinematic Symbiosis (SKS), that architecturally decouple probabilistic high-level semantic reasoning, orchestrated by Large Language Models (LLMs) acting as autonomous agents, from low-level, Lyapunov-certified deterministic execution. Furthermore, we formalise the evaluation pipeline for deployment realities, recommending a shift from empirical success rates to mathematically bounded frameworks such as Prediction-Powered Inference (PPI) to ensure robust sim-to-real generalisation. Ultimately, this review provides a rigorous technical roadmap for bridging the semantic-kinematic divide. By integrating cognitive adaptability with rigorous physical constraints, we aim to ensure that the next generation of embodied AI achieves human-level intelligence while strictly meeting the safety, accountability, and regulatory requirements for dependable clinical and industrial deployment.
A. Ebada, Yasmeen Abu-Seif, Hrushikesh Pardeshi et al.· Computers, Materials & C...· 0 citations
Results demonstrate that model-external structured state maintenance and closed-loop agentic decision making can effectively extend the local control capabilities of WAMs into embodied task execution that is plannable, verifiable, and recoverable.
Zhaopeng Gu, Bingke Zhu, Tianxin Lin et al.· 0 citations
End-to-end models that map multimodal inputs directly to future trajectories/maneuvers have emerged as an increasingly prominent research paradigm in autonomous driving. This class of models includes both Vision-Language-Action models and trajectory-generative planners. Unlike classic machine learning applications, autonomous vehicles operate in safety-critical and interaction-intensive environments where traditional open-loop imitation of expert demonstrations is not sufficient to ensure reliability. In particular, small execution errors can accumulate over time, while recovery behaviors are scarce in training data. In addition, long-horizon objectives such as safety and driving comfort are not captured by pointwise labels either. These limitations have motivated a shift toward post-training techniques, which further refine driving policies beyond pure imitation. This survey presents a unified view of post-training for autonomous driving by defining its scope and organizing the existing literature into four major families based on the form of supervision they use. For each family, we discuss its capabilities, limitations, and open challenges. We aim to facilitate a systematic understanding of this emerging area and stimulate future research on reliable and efficient post-training for autonomous driving.A collection of related papers is available at https://github.com/RYNing/Awesome-Post-Training-In-Autonomous-Driving-Papers.
Ruining Yang, Muxing Wang, Yixiao Chen et al.· 1 citation
