Social Feature Integration for Entertainment Hoax Detection: Machine Learning and DistilBERT-Fusion

Abstract

Detecting entertainment hoaxes remains a pressing challenge in the social media era, where short texts, provocative headlines, and rapid dissemination complicate verification. This study compares the effectiveness of classical machine learning models and the transformer DistilBERT, both with and without the integration of social features. The modified GossipCop dataset, consisting of 22,140 entries, includes news titles as the primary text representation and social features such as tweet count, tweet density, and viral indicators. Text was represented using TF-IDF for classical models and DistilBERT tokenization for the transformer, with performance evaluated under stratified 10-fold cross-validation. Results show that incorporating social features consistently improves classical models, with XGBoost + Social achieving the best performance (PR-AUC 0.84; F1-score 0.76), surpassing DistilBERT-Fusion (PR-AUC 0.79). McNemar’s test confirmed significant differences in error distributions, strengthening the reliability of these findings. To reduce reliance on a single empirical dataset, the social feature integration pipeline was additionally validated on the PolitiFact benchmark from a different domain, where incorporating minimal social signals yielded measurable and statistically significant gains over text-only models (Wilcoxon p = 0.0137), confirming the robustness of the social feature effect beyond the entertainment domain. Overall, the results highlight that for short-text entertainment news under the conditions examined in this study, boosting models enriched with social signals can outperform transformer-based approaches. While DistilBERT-Fusion provided competitive results, its improvement over text-only DistilBERT remained limited due to the short-text nature of the dataset, indicating opportunities for richer fusion strategies in future research.