Towards a virtual human improv troupe for social skills training

Abstract

Social skills are often practiced through role-play, where learning depends not only on producing appropriate responses but also on interpreting another person’s intentions, goals, and behavioral shifts over time. Prior research has shown the potential of large language models for social skill training, but existing approaches often emphasize rehearsal or negotiation-oriented interactions and provide limited support for open-ended social scenarios. In this work, we present the Virtual Human Improv Library, a multi-agent framework for social skill training through improvised interpersonal role-play, in which users interact with virtual characters in real time. Our approach consists of a three agent architecture: an Orchestrator LLM for scenario co-construction with a participant, a Director LLM for improvisation guidance, and an Actor LLM for in-character dialogue generation. The framework is inspired by an approach to Stanislavsky’s Active Analysis, which structures interaction through impelling actions, counteractions, tactics, and counter-tactics. We ultimately envision a Virtual Human Improv Troupe with embodied virtual actors, but in this paper we present an exploratory study of audio-only interaction between virtual actors and human users. This work contributes to the design of multi-agent role-play systems for social skill training and points toward future integration with virtual humans to teach Theory of Mind skills in socially rich interactions.