Conditional Standard Errors, Posterior Standard Deviations, and Conditional Standard Errors of Prediction in IRT Scoring

Abstract

Operational testing programs often rely on different uncertainty summaries to answer different inferential questions. This article distinguishes three such quantities in item response theory (IRT) scoring: the conditional standard error (CSE), which summarizes variability of a reported estimate under repeated measurement at fixed proficiency; the posterior standard deviation (PSD), which summarizes uncertainty about proficiency conditional on observed responses; and the conditional standard error of prediction (CSEP), which summarizes uncertainty in a future reported estimate under specified future administration, scoring, and reporting conditions. The central argument is that uncertainty summaries should be defined by their random quantities, conditioning, and intended uses, with estimation guided by those definitions. CSEP receives particular attention because it is less commonly discussed in operational IRT scoring. Building on classical test theory's standard error of prediction and Bayesian posterior predictive ideas, CSEP is defined as the standard deviation of the posterior predictive distribution of a future reported estimate produced by the operational reporting rule. A posterior predictive Monte Carlo/parametric bootstrap procedure is described for estimating CSEP under arbitrary scoring and reporting rules, with a nested law‐of‐total‐variance extension for decomposing predictive uncertainty into future measurement variability and propagated proficiency uncertainty. An appendix illustrates CSEP estimation and decomposition.