Selective Arabic Cyberbullying Detection with Multi-View Modeling, Risk Triage, and Human Deferral

Abstract

The identification of Arabic cyberbullying is generally perceived as a closed-set categorization challenge, but moderation tools need to address cases that are ambiguous, cases involving domain change in situations when review priorities are not equal. Current research introduces a selective moderation approach which can produce only either a decision about cyberbullying or refer the case to human reviewers with an optional high-priority triage flag of abusive post. As a main experiment employs a public Arabic cyberbullying corpus that was transformed into 13230 logical records and subsequently reduced to 12961 unique normalized texts after deleting 269 duplicate records (6739 cyberbullying and 6222 non-cyberbullying). Word unigram/bigram and character n-grams 3–5 were organized into TF-IDF representation, concatenated, and classified using the class-balanced LinearSVC. The results of five-fold stratified cross-validation show F1=0.9914±0.0013, macro-F1=0.9911±0.0014 and ROC-AUC=0.9996±0.0001 (mean±SD). Eventually, there was an uncertainty reject option that forwarded all cases where there was no high confidence to human agents. The deferral of 5% of the cases leads to capturing 85.7% of the errors made in non-deferral and improves the retained case’s F1 to 0.9987; a deferral of 10% of the cases captures 94.6% of the errors made and results in a retained-case F1 of 0.9995. Additionally, two audits are used to avoid inaccurate interpretations of the within-corpus score. The first involves training the cyber-bullying corpus and testing on 4,000-comment offensive language dataset of MPOLD which reduces the F1 score to 0.3882 (ROC-AUC = 0.7318). The second audit uses a different subtype proxy of hate speech from the MPOLD text allowing to obtain F1= 0.5771 and ROC-AUC = 0.7996 while addressing offensive items. Thus, it provides evidence of a necessity of human prioritization instead of autonomous risk making decisions.