Curated Semantic Mutants: Multi-purpose Artifacts for Grading and Hinting Student Test Suites
TL;DR
A human-LLM workflow that pairs each curated semantic mutant with an instructor-approved seed phrase for an on-demand LLM expansion, which shows that the curated semantic-mutant set contains a fraction of the mutants a traditional mutation engine produces.
Abstract
Mutation testing can evaluate a student's test suite, but traditional mutation tools do not encode which generated faults a course rubric should weight or what feedback should be delivered to students that do not detect a mutant. We report a human-LLM workflow that pairs each curated semantic mutant with an instructor-approved seed phrase for an on-demand LLM expansion. Deployed across two semesters of CS 3100, an introductory software-engineering course (N=1,893 submissions), the workflow provided automated per-submission feedback about undetected curated faults. In a deployment with on-demand hint expansions, 46% of students requested at least one, at roughly three cents each. At the failing submission--unit-pair level, requested expansions were associated with improvement on the next submission at 1.62 times the rate of pairs with no request, though because requests were self-selected this is not a causal estimate. A same-code comparison also shows that the curated semantic-mutant set contains a fraction of the mutants a traditional mutation engine produces. We do not compare this LLM-assisted approach with an instructor curating and writing mutants and hints by hand; whether it is cheaper or better remains an open question.
