Cross-Corpus Generalization and Calibration of Deep Learning

Abstract

Deep learning detectors for online misinformation report near-perfect in-domain accuracy, yet deployment faces distribution shift. This study asks whether such scores reflect a transferable ability to separate true from false content, auditing across three public corpora needing no social-media interface: LIAR statements, McIntire news, and ISOT news. A controlled ladder (term-frequency linear classifiers, a bidirectional LSTM, and fine-tuned DistilBERT and RoBERTa) is evaluated in-domain and across every corpus pair, with three-seed statistics, calibration, and interpretability. Both news corpora are nearly saturated in-domain (RoBERTa 0.999 ROC-area under the curve), while the short-statement corpus is hard for every model (0.657). Transfer is governed by genre: news-to-news transfer is strong (0.943) and statement-to-news is recovered by pretraining (0.935), but transfer into the statement corpus stays at chance (at most 0.527). Vocabulary overlap tracks these outcomes, and linear detectors grow overconfident under shift. Therefore, in-domain success gives weak assurance against distribution shift.