RubricAdapt-LLM: Measuring Criterion-Level Adaptation and Score Shifts Under Alternative Grading Rubrics

Abstract

Background: Large language models are increasingly used for rubric-based grading, but it remains unclear whether they adapt selectively when criterion weights change while the response, criterion definitions, and total score remain fixed. This study examined whether alternative point allocations produce targeted criterion-level adaptation or broader grading instability. Methods: A controlled paired design was applied to 1000 student responses, comprising 500 technical and 500 argumentative answers. Eight local open-weight LLMs evaluated every response under two analytic rubrics totaling 10 points. One point was transferred from Clarity to Completeness for technical responses and from Clarity to Dialecticality for argumentative responses. Of 16,000 expected evaluations, 15,999 were structurally valid, yielding 7999 complete cross-rubric pairs. Model outputs were analyzed for total-score shifts, affected-criterion adaptation, stability of unaffected criteria, model–human alignment, and correspondence with differences between two human evaluation conditions. Results: All models assigned lower mean scores under Rubric B, with mean shifts ranging from −1.239 to −0.182 points. Adaptation mechanisms differed substantially across models and response types. gemma3:4b frequently preserved technical total scores through compensating criterion changes, whereas llama3.1:8b showed extensive spillover into unaffected criteria. The Qwen models generally produced smaller total-score reductions and greater stability in unchanged dimensions. The mean score was 0.637 points higher under the human Rubric B condition than under the human Rubric A condition, although the two conditions were applied by different evaluator pairs; every model shifted negatively, and the lowest overall shift error was obtained by qwen3:4b at 1.498 points. Conclusions: Rubric sensitivity did not consistently imply localized or criterion-consistent adaptation. Reliable evaluation of LLM graders therefore requires separate analysis of total scores, affected criteria, unaffected criteria, human alignment, and cross-condition shift correspondence.