A multidimensional Elo method for online item calibration

Abstract

As adaptive testing and learning systems proliferate, efficient item calibration is a pressing need. The Elo rating system jointly estimates student ability and item difficulty, offering an alternative to item response theory (IRT) models. Yet existing multidimensional extensions have mainly focused on ability tracking rather than item calibration. This study proposes the multidimensional Elo (MELO) method for online item calibration when no calibrated item pool or anchor items are available. Two simulations (500 replications per condition) evaluated accuracy via RMSE and conditional bias. Study 1 compared MELO with multidimensional IRT (MIRT) estimation across dimensional structures (between-item vs. within-item), inter-dimensional correlations, and sample sizes. Study 2 manipulated dimensional structure, adaptive selection proportion, and sample size–test length combinations at a fixed number of responses. An empirical application compared MELO and MIRT item difficulty estimates using real response data. For item difficulty, the accuracy gap between MELO and MIRT was modest and largely confined to extremely difficult items; under within-item multidimensionality with strong inter-dimensional correlation, MELO was significantly more accurate. Conditional bias exhibited a regression-to-the-mean pattern concentrated at the distribution extremes. Adaptivity generally benefited within-item multidimensionality, whereas under between-item structures only partial adaptivity following a random item-selection phase was beneficial, and fully adaptive selection induced substantial item exposure imbalance and degraded calibration. The effect of the sample size–test length configuration on item calibration was moderated by dimensional structure and adaptive selection proportion. Longer tests administered to fewer examinees improved ability estimation. In the empirical data, MELO and MIRT item difficulty estimates correlated almost perfectly ( r = 0.997). The findings support MELO for preliminary item calibration under the conditions examined.