SCAR-FND: Auditing Lexical Shortcuts in the ISOT Fake News Benchmark Through Automatic Discovery and Matched Ablation

Abstract

Very high scores on fake-news benchmarks can be misleading when artifacts introduced during dataset construction are strongly tied to the class labels. This study presents SCAR-FND, a source-confound-aware auditing framework that identifies label-associated lexical shortcuts from training data and then measures how strongly a classifier depends on them. The ISOT corpus initially contained 44,898 articles. After removing 5,798 exact normalized duplicates, 39,100 unique articles remained (21,195 real and 17,905 fake). As a simple dataset diagnostic, the presence of Reuters alone classified 99.37% of the cleaned corpus correctly; Reuters appeared in 99.82% of real articles but only 1.17% of fake articles. A TF-IDF/XGBoost baseline achieved 99.69% accuracy (F1 99.66%; MCC 0.9938), whereas masking explicit Reuters/source cues reduced accuracy to 97.86%. Candidate shortcuts were then ranked on the training split using the absolute difference in class-conditional document frequency. Reuters was automatically ranked first, with a leakage score of 0.9865. Removing the top 100 shortcut tokens, together with any TF-IDF unigram or bigram containing them, reduced accuracy to 93.76%, F1 to 92.99%, and MCC to 0.8749. By comparison, matched random removal of the same 811 TF-IDF features across three repetitions preserved a mean accuracy of 99.68%, producing a targeted-ablation gap of 5.92 percentage points. These findings indicate that ISOT contains a small but highly predictive set of lexical, source, editorial, and stylistic cues. SCAR-FND therefore complements conventional accuracy reporting with a transparent audit of what information the classifier is using.