Grading scale impact on LLM-as-a-judge: human-LLM alignment is highest on 0–5 grading scale
Large language models (LLMs) are increasingly used as automated evaluators, yet prior works demonstrate that these LLM judges often lack consistency in scoring when the prompt is altered. However, the effect of the grading scale itself remains underexplored. We study the LLM-as-a-judge problem by comparing two kinds of...
Weiyue Li, Minda Zhao, Weixuan Dong et al.
· npj Artificial Intelligence · 0 citations