CARE: A Calibration-Aware Framework for Cyberbullying Detection Under Distribution Shift

Abstract

Cyberbullying detectors are often evaluated by in-domain accuracy, but real moderation systems need reliable probabilities under distribution shift. This paper presents CARE, a calibration-aware and leakage-controlled evaluation framework. CARE compares four compact encoders with two LoRA-adapted 7B language models on the SynBullying corpus. The protocol uses conversation-level splits, speaker anonymisation, three-seed averaging, cluster-bootstrap testing with Holm–Bonferroni correction, and five-fold GroupKFold robustness checks. The six systems achieve similar in-domain Macro-F1 near 0.91, and no tested contrast against RoBERTa-base survives Holm–Bonferroni correction. Out-of-domain behaviour differs on OLID, HateXplain, and Civil Comments. On Civil Comments, the worst-to-best Expected Calibration Error ratio between the encoder and LoRA-adapted language-model panels reaches 14.9 times. After temperature scaling, the residual ratio remains 9.4 times. We call this family-level discontinuity the Calibration Cliff. A synthetic-only retraining study on all ten authentic conversation clusters and an external authentic benchmark confirm the cliff’s direction in every matched-seed comparison, while both families lose accuracy under register shift. CARE further introduces a lightweight confidence-routing policy, executed end-to-end at every reported operating point. At threshold 0.90 the policy reduces mean out-of-domain calibration error by 22 percent at roughly half the pure language-model latency on commodity hardware, and it automatically routes several times more traffic to the language model under distribution shift than in-domain. Under the evaluated protocol, the value of LoRA adaptation lies less in in-domain accuracy than in cross-distribution robustness and probability reliability. CARE provides an evaluated accuracy, calibration, and latency trade-off for real-time cyberbullying moderation.