Exploration of Suboptimal Modelling Choices—Ordinal Modelling as a Way to Better Measure Effect Size Heterogeneity?

Abstract

Heterogeneity in population effect sizes has often been suggested as impairing replication success. The validity of this line of argumentation rests on the assumption that reported heterogeneity estimates provide an accurate description of effect size heterogeneity. However, efforts to precisely measure between-study heterogeneity may be affected by inadequate model specification. Most primary analyses in psychology treat Likert-scale data as continuous rather than ordered-categorical, thereby applying models that are, strictly speaking, mis-specified. This can lead to various statistical issues like inflated error rates and distorted estimates which introduce biases in the heterogeneity estimation. To assess the impact such model misspecifications have, 16 effects with a two-group experimental design and a single-item DV from multi-lab replication projects Many Labs and Psychological Science Accelerator with a total of 94.863 participants clustered in 644 single replication sites were reanalysed with Bayesian ordinal regression models. The reanalysis allows for a comparison between heterogeneity estimates obtained using either the ordinal or standard linear model which treats Likert scales as continuous measures. The comparison of heterogeneity estimates between the two modelling approaches showed only small differences. However, generalisability is limited as some of the assessed effect sizes are close to zero and show little heterogeneity. Based on these results, further implications on the role of model choice for better assessment of heterogeneity are discussed.