Structured Representations For Human-Robot Interaction with Non-Expert Users

Abstract

A prerequisite for deploying robots alongside people is the ability to exchange information fluidly across multiple modalities: robots must interpret implicit social signals, receive explicit instructions, and communicate back in ways non-experts can understand. A key challenge is that humans naturally exchange information through multiple modalities, requiring computational systems to consider heterogeneous signals to be fully fluent, interactive partners. My key insight is that structured intermediate representations can serve as a common backbone across modalities, grounding both understanding and communication without requiring specialized user input. I develop two such representations: a topological formalism for reasoning about multimodal social signals in human environments, and a motion descriptor framework that translates natural language stylistic intent into steerable robot behavior, supporting bidirectional refinement through grounded feedback. Together, these contributions advance a vision of robot systems that that fluently share spaces and interactions with any user.