From Supervised Baselines to Positive–Unlabeled Learning: Fake News Detection on the Truthseeker2023 Ground-Truth

Abstract

Publication of fake/false news on social media exceeds the controlled ability of manual fact-checking, and the labeling of operations is often not complete and trustworthy. In this research, We explore twitter level truthfulness classification on Truthseeker 2023 ground truth corpus, and use a hybrid feature representation that incorporates textual features as TF-IDF(Term Frequency-Inverse Document Frequency) vector, together with pre-calculated user, content and engagement features. The baseline experiments include initial experiments of standard supervised models, such as, Logistic Regression, Random Forest, Support Vector Machine, XGBoost, and LightGBM as well as a soft-voting ensemble, on a subset of a held-out test split. These findings show that linear and boosted -tree models have strong discriminatory capability, and the ensemble achieved an accuracy of 93.73 and ROC-AUC of 95.39. We then learn using the label scarce regime through Positive Unlabeled (PU) learning where a fraction of the target class is labeled and the rest are explicitly unlabeled mixture. In the PU model, Elkan-Noto approach presents the highest level of efficiency, with an accuracy of 84.53 per cent and an ROC-AUC of 90.40; a Two-step PU strategy presents a higher- recall option. All in all, the results measure the performance loss to be expected in passing between fully supervised models to PU learning and show that even with the necessary calibration of PU strategies (semi-supervised learning), a significant percentage of the intrinsic signal can be recovered without the need to explicitly label examples as negative.