A Hierarchy of Inferability: The DIRECT Framework for Defining Human involvement in AI-Assisted Identification and Rating of Perceptual Indicators in Open Teacher Discourse

Abstract

This study examines the conditions under which generative AI (GenAI) can support the identification and rating of perceptual indicators related to technology adoption in open-ended teacher discourse. Grounded in the Technology Acceptance Model (TAM), the study conceptualizes three levels of inferential demand: direct-evidence, synthesized-evidence, and TAM construct interpretation. Data from 174 K-12 teachers were analyzed through a human-AI comparative design using weighted Cohen’s kappa, before and after a structured training process. Findings indicate that agreement between GenAI and human raters varies systematically with inferential demand, with stronger alignment for direct-evidence indicators and more limited reliability & validity for higher-level, theory-driven constructs. Training improved alignment primarily for lower- and mid-level variables, while highlighting the role of “Insufficient Information” as a meaningful analytical outcome. The study introduces the DIRECT framework, which links inferential demand to the distribution of human-AI roles, offering a conceptual and practical guide for integrating GenAI into identification and rating of perceptual indicators.