Correct Verdicts, Wrong Field: Attribution and Capability Are Dissociable When Small Language Models Follow a Rubric

Abstract

A behavioral format assay changes the structure of a prompt and reads whether a model’sanswers improve. Verdict accuracy is the usual measure. We show that it is not sufficient,and that a model’s self-reported basis for its answer is not sufficient either. We gave threelocal language models a structured decision rubric that names one field, a scope condition,as the field on which the verdict turns, and asked each for a verdict and for the field thatdetermined it. Across six decision classes and two scenario polarities, verdict accuracy sat ator near ceiling (0.96–1.00), while the share of correct verdicts that cited the scope field variedwidely by model, from 0.04 to 0.66. On the firing scenarios a 32B model cited the scope fieldin none of its 48 correct verdicts, naming the Marker axis (a recognition axis: an invariantdescribing what a violation looks like) instead; the thinking model named the scope field in15 of 48. The non-scope citation is model-specific: the Marker axis for the two Qwen models,and the Pull character (the field for why the locally-wrong action is attractive) for Mistral.Yet the cited field is not the reasoning: in all 33 of the thinking model’s correct base runsthat cited a non-scope field, the recorded reasoning trace engaged the scope condition andthen named a different field. The self-report is unfaithful. A grader scoring only the verdict,or trusting the cited field, would misread both. We then removed the two recognition axes.Accuracy did not fall, but it was at ceiling in both arms, so the accuracy test of fragilityis uninformative; what moved was the citation, toward the scope field (the thinking modelto 96 of 96) and, for the 32B model, toward the surviving Rest axis (the field for territorywhere the decision class does not arise; scope 65, Rest 30 of 96). The named field trackswhich fields are present, not a fixed dependency. Attribution is dissociable from capability:the field a model reports is neither the field its reasoning used nor the only field it can name.We argue that a format assay must not read the verdict or the model’s self-reported fieldas evidence that the intended field did the work; the self-report is checked against the traceand against an intervention, not trusted.