JudgeProfile: Understanding and Steering Subjectivity in LLM Judges
JudgeProfile is introduced, a framework that dissects LLM evaluation into perception (how a judge compares two responses across specific attributes like clarity, correctness, and detail) and prioritization (how much each attribute influences the final choice) and finds a hidden consensus in perception.