greCAPTCHA: Assessing Understanding as Evidence of Research Authorship Under Generative AI
TL;DR
Initial evidence is provided that greCAPTCHA can assess manuscript-specific understanding under proctored conditions and the appropriate construct validity for author understanding is remarked on, while also suggesting important changes to be made before deployment.
Abstract
Conferences, journals, funders, schools, and universities are struggling with a surge of potentially AI-generated submissions from ostensibly human authors, who may not have exercised sufficient human oversight for their manuscripts. In turn, institutions evaluating submissions can no longer reliably credit expertise based solely on authors'names on submitted work. To address this problem, we propose greCAPTCHA, a proctored assessment approach that measures authors'understanding of research manuscripts via the construct of capacity to verify, which we define as the knowledge and reasoning required to critically assess the contents underlying one's contributions to a manuscript. greCAPTCHA generates questions assessing multiple levels of understanding and provides an evaluative report based on authors'responses. Using a prototype implementation, we conduct a user study and semi-structured interviews with $31$ researchers to evaluate greCAPTCHA. Its automated scores predict which papers were or were not authored by study participants with an AUC of $0.90$. Participants reported positive overall experiences with the system and remarked on the appropriate construct validity for author understanding, while also suggesting important changes to be made before deployment. Our results provide initial evidence that greCAPTCHA can assess manuscript-specific understanding under proctored conditions.
