The Use of Artificial Intelligence Tools in Assessing Content Validity: A Comparative Study with Human Experts
TL;DR
Findings indicate that AI tools can complement expert judgment in content validity assessment when methodological control is ensured and highlights the potential of hybrid human–AI frameworks in educational measurement, while emphasizing the importance of model transparency and context management.
Abstract
This study investigated whether artificial intelligence (AI) systems could evaluate the content validity of B1-level English reading comprehension items in a manner comparable to human experts. A 25-item multiple-choice test was developed and rated by four human experts and four AI models (GPT-4o, Gemini 2.0, Replika, and Chatfuel). To ensure methodological transparency, a standardized few-shot prompting framework was employed. Each model was accessed in a new session to avoid context drift, and identical prompts were provided to maintain consistency across evaluations. The Content Validity Ratio (CVR) and Item Content Validity Index (I-CVI) were calculated and compared using the Wilcoxon Signed-Rank Test. The results revealed no statistically significant difference between AI- and human-generated scores, suggesting that AI evaluators produced ratings similar to those of human experts. Although fine-tuning could not be performed due to the closed-source nature of the models, standardized prompting procedures enhanced reliability and replicability. These findings indicate that AI tools can complement expert judgment in content validity assessment when methodological control is ensured. The study highlights the potential of hybrid human–AI frameworks in educational measurement, while emphasizing the importance of model transparency and context management.