Can We Trust This Evaluation? Instance-Level Reliability of LLM-as-a-Judge
LLM-as-a-judge evaluation commonly aggregates multiple judges'decisions into a majority vote, yet this panel decision process does not guarantee that the evaluation of an individual item is reliable. We propose Structured Cross-fitted Uncertainty Risk (SCUR), a label-free approach for estimating instance-level evaluati...
Yang Ba, M. Mancenido, Rong Pan
· 0 citations