Automated systems for assessing student knowledge based on artificial intelligence: development and empirical verification

Abstract

The study aimed to develop a three-level conceptual model of an AI-based automated knowledge assessment system and to provide empirical evidence regarding its performance within a higher education context. For this purpose, a quasi-experimental design (pretest–posttest) was used with the participation of 250 students of humanities specialties (bachelor's: n = 163; master's: n = 87). The toolkit consisted of parallel forms of the academic achievement test (40 tasks, α = 0.83), an adapted TAM scale. For analysis, paired t-test, confirmatory factor analysis, hierarchical multiple regression and dominance analysis were used. Students demonstrated higher post-test academic achievement following implementation of the AKS model. The analysis showed a hierarchy of relative importance of levels (R² = .61): semantic-discursive level (50.8%) > personalized feedback (31.1%) > syntactic-formal (18.1%). Students positively perceived the system on all TAM dimensions: PU = 3.91, PEOU = 3.67, OS = 3.78. Therefore, the hierarchy of predictors became an important recommendation for developers of the ASZ regarding the priority of investments in the semantic level. The findings suggest that a phased implementation strategy may represent a pedagogically promising approach. The results are relevant for higher education institutions operating in a distance or blended format.