A Low-Resource Arabic Dataset and Transformer-Based Benchmark for Dark Pattern Detection in E-Commerce Mobile Applications
TL;DR
Per-class analysis shows every model struggles with several minority categories, indicating Arabic dark pattern detection remains genuinely difficult at current data volumes, indicating pre-training on dialectal, code-switched Arabic may matter more here than Arabic pre-training alone.
Abstract
Arabic remains underrepresented in many task-specific natural language processing (NLP) resources and benchmarks, particularly for specialized user interface (UI) understanding tasks. In mobile commerce, Arabic UI text may contain persuasive or deceptive design cues known as dark patterns; however, Arabic-language dark pattern detection remains largely unexplored. To the best of our knowledge, this paper presents the first ML-based benchmark for Arabic dark pattern detection. We construct a novel annotated dataset of 223 Arabic UI text strings from nine e-commerce mobile applications operating in Saudi Arabia, labeled across five dark pattern categories and a non-dark-pattern class (Cohen’s kappa κ = 0.89). Using a stratified, leakage-free 70/10/20 split with parent-aware paraphrase augmentation applied only to the training partition, we fine-tune five pretrained transformer models: AraBERTv2, MARBERT, mBERT, BERT-base-uncased, and RoBERTa-base. Our primary evaluation is 5-fold cross-validation on the 223 original, non-augmented instances, separate from the augmented training corpus used for the held-out test comparison. Under this evaluation, MARBERT achieves the strongest performance (mean macro-F1 = 0.4230), numerically ahead of AraBERTv2 (0.2998) by a margin that does not reach statistical significance at five folds (p ≈ 0.064), and ahead of mBERT (0.3234); MARBERT significantly outperforms both English-only baselines, and mBERT is numerically stronger than both, though mBERT was not directly tested against them for significance. This suggests pre-training on dialectal, code-switched Arabic may matter more here than Arabic pre-training alone. Per-class analysis shows every model struggles with several minority categories, indicating Arabic dark pattern detection remains genuinely difficult at current data volumes. The annotated dataset is publicly released to support future low-resource Arabic NLP research.