GESR: Graph-Guided Evidence-State Repair for Citation-Grounded Multimodal Fact-Checking

Abstract

Multimodal fact-checking systems are increasingly expected to return not only a veracity label but also a citation-grounded re port. Yet a correct label may still be accompanied by query drift, hallucinated citations, unsupported evidence statements, or over strong conclusions. We argue that many such failures arise from evidence-state management errors in two local windows: retrieval plan formation before evidence retrieval and citation integration after evidence retrieval. We propose GESR, a training-free inference time framework that uses an executable claim-local evidence-state graph to constrain local repair actions under a fixed retrieval budget with one real retrieval call per example. Unlike graph-as-context prompting, GESR does not merely expose graph information to the generator. It uses graph checks to accept or reject typed actions, including entity, time, source, and modality edits before retrieval, and citation replacement, claim weakening, modality-support inser tion, and verdict downgrading after retrieval. To evaluate citation grounding, we build MR2-E, a controlled evaluation protocol on top of MR2 with a frozen retrieval index, opaque citation identifiers, top-5 citation resolution, and sentence-level support judgments. On MR2-E under the fixed single-retrieval-call setting, GESR shows statistically inconclusive label-level F1 differences but larger and more stable improvements in evidence consistency, citation accu racy, and human-judged citation grounding than verifier-repair and graph-exposure baselines.