Type I Error and Statistical Power of Welch’s t-Test Under Multiple Assumption Violations

Abstract

This study employed Monte Carlo simulations to examine the effects of nonnormality, heteroscedasticity, total sample size, and unequal group sample sizes on the Type I error rate and statistical power of Welch’s t-test. The results showed that, consistent with previous studies, Welch’s t-test maintained stable control of the Type I error rate under conditions combining heteroscedasticity and unequal group sample sizes. However, when skewness and heteroscedasticity occurred simultaneously, the Type I error rate increased substantially. Under some conditions, the Type I error rate even increased as the total sample size grew. Statistical power generally improved with increasing total sample size, but varied considerably depending on the direction of heteroscedasticity, even when the total sample size remained the same. The effect of kurtosis was relatively small. These findings indicate that although Welch’s t-test is relatively robust under normality, it may fail to maintain adequate control of the Type I error rate even with large samples when skewness and heteroscedasticity are simultaneously present. Possible explanations for the increase in Type I error rates under large-sample conditions are discussed. The findings also highlight the need for caution when applying and interpreting Welch’s t-test under complex violations of assumptions involving skewness, heteroscedasticity, and sample-size configurations.