MedJudge: Medical Multimodal Reward Modeling

Abstract

As large multimodal models (LMMs) become increasingly integrated into clinical workflows, robust evaluation and alignment mechanisms are critical to ensure their reliability in decision support. Existing reward models (RMs) typically reduce preference supervision to scalar scores, limiting transparency and diagnostic insight. In this work, we introduce MedJudge, a multimodal medical reward modeling method that supports interpretable, evidence-grounded, and clinically-aligned decision evaluation. Given a clinical query (text and image) and candidate responses, MedJudge outputs both strict pairwise preferences and concise rationales, enabling transparent verification and inference-time reranking. To overcome the lack of high-quality supervision data, we construct a large-scale synthetic dataset with five complementary generation heuristics, and compare three training paradigms—scalar (BT), supervised (SFT), and reasoning-augmented SFT (SFT-R)—across 8 open-source models. Our comprehensive experiments show that while all methods improve over baseline, SFT-R provides comparable accuracy with richer explanations, and BT achieves the strongest overall performance. Beyond decision accuracy, we propose UMLS-based Concept Overlap (UCO) to evaluate explanation quality, measuring concept-level alignment with clinician expectations. Compared to BERTScore, UCO better reflects human preferences across multiple statistical tests. We further show that our BT-trained judges improve Best-of-4 answer selection by up to 2.75%, demonstrating practical utility in clinical inference-time decision reranking. Our results establish MedJudge as an effective, interpretable, and clinically grounded framework for multimodal medical reward modeling and evaluation. Our code, data and scripts are available at https://github.com/yunhonghe1021/MedJudge.