Same construct, same conclusion? A construct-observability perspective on self-report and non-self-report indicators in AI education research: a two-phase systematic review

Abstract

AI-enhanced education research frequently combines self-report with tests, ratings, traces, or other non-self-report (NSR) indicators, but the two methods need not yield the same reported outcome conclusion. We conducted a two-phase systematic review. Phase 1 coded measurement methods and constructs in 2,099 title/abstract records. Phase 2 used a purposive, construct-stratified full-text sample and retained 51 same-construct entries from 47 reports, including inferential, descriptive, and qualitative evidence. A mixed-evidence descriptive synthesis classified 30 entries as concordant, 11 as discordant, and 10 as mixed across dimensions, conditions, or time points. The full-sample LOW-versus-HIGH comparison was not significant (p = 0.555); a restricted sensitivity analysis of 10 entries from 10 reports with sufficiently aligned paired inferential evidence was likewise non-significant (p = 0.524; nine classified entries tested). The descriptive LOW-HIGH ordering reversed between analyses, precluding a stable directional claim. Common-context within-study patterns suggested that output externality, feedback immediacy, and criterion explicitness may help explain heterogeneity, but they do not establish a causal mechanism or universal hierarchy. We develop construct observability as a testable, hypothesis-generating framework for organising cross-method heterogeneity. The practical implication is conditional: measurement should follow the claim. Self-report can be appropriate for perceptions, confidence, experiences, and self-observed behaviours; claims about demonstrated competence benefit from directly aligned performance-related evidence, while such NSR indicators also require their own validity justification.