With the advancement of Generative Artificial Intelligence (GenAI) and in particular Large Language Models (LLMs), increasing focus has been placed on the development of conversational agents across domains, including education. Within education research, pedagogical agents (PAs) have traditionally been designed by researchers, producing positive results when implemented in ways consistent with good pedagogical practice. Findings suggest that the effectiveness of PAs depends on the degree to which learners relate to their agents and perceive them as socially and behaviorally realistic. One way to accomplish this is to include end users in the PA design process. Authoring tools have lowered technical barriers, enabling novice designers to participate in educational technology design. Building upon this work, we introduce the Pedagogical Agent Toolkit (PATK), an extendable and modular toolkit with AI-based features with the purpose of expanding novice designer agency and opportunity in the creation and customization of PAs. This paper outlines the design principles, architecture, and features of the PATK.
Samuel Hum, Jennie Lee, Jessica R. Gladstone et al.· 0 citations
Large language models (LLMs) now power the reasoning core of intelligent virtual agents deployed across an expanding range of social settings, from tutoring students and supporting patients in healthcare, to mediating group discussions and representing humans in various social settings. Effective deployment demands social cognition, the capacity to model what others believe, detect deception, and coordinate strategic action under incomplete information. These capacities, exemplified in the social dynamics of the game Among Us, remain poorly characterized in current LLM evaluation frameworks. We introduce a strategic game arena that situates LLM agents in social deduction scenarios inspired by Among Us, requiring theory of mind, deception detection, and cooperative deliberation under uncertainty. We evaluate 19 open-weight models across 10,134 games and 289,614 utterances, testing both homogeneous and heterogeneous crews. Our experiments reveal three findings. First, crewmates voting through generative reasoning reach only \(50.4\% \pm 4.4\%\) F1 when identifying imposters, while a logistic regression classifier trained on the same discussion transcripts achieves \(85.3\%\) F1. Second, scaling model parameters yields a statistically significant but practically marginal improvement. Medium models (60–82B) reach \(52.8\%\) F1 against \(46.1\%\) for small models (7–20B), a 6.7 point gain (Mann–Whitney U, p = 1.5 × 10− 34). Third, agents fail to integrate evidence coherently during deliberation. Imposters self-incriminate in \(4.12\%\) of their statements, yet crewmates eject the confessing agent only \(33.8\%\) of the time. Crewmates reverse their stated suspect between consecutive rounds without new justification in \(41.9\%\) of cases. Sentiment remains uniformly neutral whether an agent is reporting a body or delivering a routine update. These gaps identify concrete limits in the social cognition of current LLM-powered agents and motivate architectural changes for virtual agents that must cooperate with humans. The source code and the live arena viewer are available at https://ufdatastudio.com/projects/agents-among-us.
Kevin Kurian, Kevin Scroggins, Emmanuel Dorley et al.· 0 citations