KSA Profiles for Standard Setting: A Generative AI Approach to Empirically Grounded Cut Score Evaluation
TL;DR
Generative AI is applied to AP U.S. History and AP World History essay data to demonstrate how this kind of evidence could help panels evaluate whether a proposed cut score captures the distinctions a policy intends and inform discussion of proposed cut scores or performance expectations considering evidence of the KSAs students demonstrate.
Abstract
Standard setting for essay‐based assessments often relies on expert judgment and performance‐level descriptors, but these sources do not always show how examinees at adjacent score levels differ in the knowledge, skills, and abilities (KSAs) they demonstrate. This study introduces a method that uses generative AI to fill that gap. A large language model reads student essays and estimates the likelihood that each student demonstrated a set of pre‐defined knowledge, skills, and abilities (KSAs). Those estimates are then summarized as KSA profiles—representations of which competencies are characteristically more or less evident within a score region or at a proposed cut score boundary. To demonstrate the method, we apply it to AP U.S. History and AP World History essay data using hypothetical cut scores as an exploratory example. Results show that profiles differed across score levels, that the same broad KSA structure emerged across both subjects, and that AI‐derived measures aligned with established indicators of historical reasoning. We discuss how this kind of evidence could help panels evaluate whether a proposed cut score captures the distinctions a policy intends and inform discussion of proposed cut scores or performance expectations considering evidence of the KSAs students demonstrate.
