Tian-Yong Hao

BayesJudge: Uncertainty-Aware Bayesian Meta-Evaluation of Human and LLM Judgments

AI evaluation pipelines often produce conflicting judgments rather than clean labels. In pairwise LLM evaluation, this conflict is especially visible: disagreement can arise from ambiguous items, underspecified rubrics, heterogeneous or unstable human raters, or an LLM judge whose verdict changes when the response orde...