Comparing machine learning models for academic success prediction in online learning

Abstract

Early-warning models must balance the detection of unsuccessful outcomes with the workload of student support. We compare logistic regression, a decision tree, a random forest, and gradient boosting using 32,593 records from the Open University Learning Analytics dataset. Sixteen features are evaluated at 30, 60, and 90 days and over the full course using a student-grouped holdout and repeated grouped cross-validation. Gradient boosting has the highest cross-validation macro-F1; holdout macro-F1 is 0.706, 0.751, and 0.775 in the early windows. Assessment features provide substantial predictive information. Target risk recalls of 0.80-0.90 require flagging 48.3-73.3% of records with precision 0.541-0.662. These operating points are selected on holdout outcomes and require independent validation. Full-course performance is retrospective and uses a different population. Approximate statistical comparisons and unresolved feature-availability times limit inference. The results motivate a prospective evaluation of advisor support rather than the establishment of an effective deployment policy.