CGPO: Counterfactual Grounding Policy Optimization for Evidence-Sensitive Pathology Vision-Language Reasoning
Abstract
Multimodal large language models (MLLMs) have shown growing potential in digital pathology, yet their reasoning often remains weakly grounded in the actual visual evidence within images. Existing supervised fine-tuning and reinforcement learning approaches primarily optimize final answer correctness, resulting in a persistent disconnect between visual perception, evidence localization, and reasoning, thereby leaving models unconstrained to rely on linguistic priors. We propose Counterfactual Grounding Policy Optimization (CGPO), a reinforcement learning algorithm that explicitly promotes Evidence-Sensitive Reasoning (ESR) by modeling the causal dependence between model predictions and localized visual evidence under weak supervision. CGPO constructs counterfactual images by masking predicted evidence regions and leverages token-level distributional changes to derive ESR-aware response scaling and fine-grained token-level advantage modulation. A grounding consistency regularization further constrains evidence localization to prevent reward hacking during training. Experiments across multiple pathology benchmarks demonstrate that CGPO consistently improves both task performance and visual evidence localization, achieving performance exceeding strong closed-source models and approaching human experts.