Skip to content
Book Open access

When Not to Diagnose: Evaluating Evidence-Sufficiency Recognition in ECG Multimodal Language Models with ECGTrustBench

Oct 2026 · Proceedings of the 4th International Workshop on Human-Centered Sensing, Modeling, and Intelligent Systems · pp. 83-88 · 0 citations · 4 references

TL;DR

ECGTrustBench is introduced, which evaluates evidence-sufficiency recognition as answerable-versus-insufficient QA over 3,300 image-question pairs, with corruption criteria unanimously approved by three licensed clinicians.

Abstract

Mobile electrocardiogram (ECG) capture routes photographed or scanned 12-lead images to an AI model or a clinician, yet missing leads, calibration loss, or photographic degradation can remove diagnosis-critical evidence while the tracing remains visually readable. A condition-aware evidence-sufficiency check is therefore needed to flag unusable captures for recapture or clinician review before diagnosis. Existing ECG multimodal large language model (MLLM) evaluations cannot support this, as they assume every question is answerable. We introduce ECGTrustBench, which evaluates evidence-sufficiency recognition as answerable-versus-insufficient QA over 3,300 image-question pairs, with corruption criteria unanimously approved by three licensed clinicians. Across ECG-specific, medical-domain, general-domain, and proprietary MLLMs, we find a consistent gap between answering and abstaining: ECG-specific models reach up to 77.3% clean accuracy yet abstain on at most 10% of evidence-deficient cases, while proprietary models achieve recognition up to 82%. Overall, current models respond more to visible degradation than to missing diagnostic evidence, highlighting a critical gap for evidence-sufficiency gating.

Read PDF

Similar papers

Preprint Nov 2026

ECGJev: Question-Conditioned Option Scoring for 12-Lead ECG Interpretation

Clinical interpretation of a 12-lead electrocardiogram (ECG) often requires answering targeted diagnostic questions with clear, quantifiable decisions. We introduce ECGJev, the first ECG framework built on a decision decoder, a pretrained language model trained to return a probability for every listed option. A questio...

Zhuo-Yuan Li, Xi-Dong Wu, Han-Yu Cui et al. · 0 citations
#artificial intelligence Preprint Oct 2026

IECG: Weakly-Supervised Interpretability-Enhancing Foundation Model for Electrocardiogram

IECG is presented, a weakly-supervised ECG foundation model that jointly performs diagnostic classification and abnormality localization without requiring manual abnormal region annotations, overcoming a long-standing bottleneck in interpretable ECG learning.

Huynh Dang Nguyen, Trong-Thang Pham, A. Đỗ et al. · 0 citations
Open access Oct 2026

Brugada syndrome: a multicenter study on ECG image-based self-supervised learning for non-invasive prediction of sodium channel blocker response

Brugada syndrome (BrS) is an inherited channelopathy predisposing to ventricular fibrillation and sudden cardiac death. Definitive diagnosis often requires provocation with sodium channel blockers, yet this procedure carries inherent risks. Predicting challenge test outcomes from baseline electrocardiograms (ECGs...

N. Alhourani, T. Okatani, T. Watanabe et al. · 0 citations
Preprint Oct 2026

MS-ECG-FM: Towards a More Universal Electrocardiogram Foundation Model for Health Monitoring using Multi-source Contrastive Learning

Electrocardiography (ECG) records the electrical activity of the heart, aiding diagnosis by detecting abnormalities in cardiac function. ECG foundation models have demonstrated promising results, but are limited by a reliance on ECG interpretation reports as their sole supervision. Because interpretation reports only c...

Robert A. Lewis, I-Min Chiu, Kyle Verrier et al. · 1 citation
Open access Oct 2026

Exploratory evaluation of a medical vision-language model for electrocardiographic detection of acute coronary occlusion.

INTRODUCTION Vision-language models (VLMs) are increasingly explored for clinical interpretation tasks. Electrocardiographic (ECG) identification of acute coronary occlusion (ACO) is a high-risk application requiring accurate pattern recognition, and medical VLM performance on this task remains poorly characterised out...

Andrew J. Bishop, Christopher A. Bain, Benjamin N. Meadley · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.