Agreement by Construction: A Deterministic Filter Inflates Inter-Judge Reliability in LLM-Judge Evaluation

Abstract

Aim: This study investigates how a deterministic pre-judgment rule, which assigns a verdict before LLM judges evaluate a response, affects estimates of inter-rater reliability, with particular attention to Cohen’s κ. Methods: Using 600 paired judgments from the LoCoMo benchmark, we compared Cohen’s κ between two LLM judges (qwen3:8b and deepseek-r1:8b) before and after applying a deterministic refusal rule that assigned the same negative verdict to both judge columns without invoking either judge. We separately evaluated the 461 judgments the rule did not affect, examined the marginal distributions, tested three additional deterministic filters, and applied a constant-rater control. Results: Applying the refusal rule increased full-sample κ from 0.056 to 0.714. On the 461 judgments the rule did not touch, κ was 0.064 with and without the rule, so the apparent improvement did not come from greater agreement between the judges. Decomposing the 2 × 2 table showed that the marginal shift accounted for most of the increase. With the same judges raked to balanced marginals, κ was 0.454, and three other deterministic text filters inflated κ by similar amounts. On answerable questions the two judges credited a refusal as correct 84.2% and 97.1% of the time. Conclusion: When a deterministic pre-judgment rule supplies identical verdicts to both judges, full-sample Cohen’s κ can reflect the scoring protocol rather than independent inter-judge reliability. Recommendation: We recommend reporting κ on the rows the protocol does not determine, each judge’s marginal rate, and a constant-rater control. We release per-question verdicts and the analysis code.