Asymptotic Standard Error of the Classification Consistency Index Under Item Response Theory

Abstract

Classification consistency evaluates the degree of consistency of the classifications based on observed scores from repeated testing. The classification consistency index is often used in practice for tests that categorize examinees into two or more categories with respect to a specific standard. In this paper, the asymptotic standard error of the classification consistency index under item response theory (IRT) that utilizes raw scores was derived using the delta method for the three-parameter logistic (3PL) model. The derived formula was applied to real data sets, and its accuracy was examined using simulated data sets under various dichotomous IRT models, sample sizes, and latent densities. In general, the asymptotic standard errors were mostly accurate for the 3PL and 2PL models. The formula also worked well for the 1PL model if multiple cut scores were applied at the same time or a single cut score that was not close to the mean score was used to make binary decisions.