SwinAraBERT-Fusion: A Multimodal Transformer Architecture for Robust Arabic OCR and Handwriting Recognition with Diacritic-Aware Contextual Modeling

Abstract

Arabic Optical Character Recognition (OCR) and Handwriting Recognition (HWR) remain persistently challenging due to the script's inherently cursive nature, context-dependent positional letter forms, dense diacritic system (tashkeel), and high inter-class visual similarity among characters. While transformer-based architectures have driven significant advances in text recognition, most existing Arabic OCR systems rely on unimodal visual encoders that underutilize the rich morphological and syntactic structure of the Arabic language. This paper proposes SwinAraBERT-Fusion, a novel multimodal deep learning architecture that integrates a SwinV2 hierarchical vision transformer with an AraBERT contextual language model through a cross-attention fusion mechanism augmented with adaptive gating. The system is designed for line-level recognition of both printed and handwritten Arabic text, with explicit handling of diacritics. Training employs a two-stage curriculum learning strategy on combined benchmark datasets (KHATT, IFN/ENIT, APTI, SARD). Preliminary experiments on "Qalam_Net_V2" baseline, a fine-tuned Transformer-based architecture built on TrOCR model using 500 test samples yielded a Character Error Rate (CER) of 10.87% and Word Error Rate (WER) of 56.20%, revealing critical failure modes - diacritic substitution and short-word sensitivity - that directly motivate our multimodal design. The proposed architecture is expected to significantly outperform unimodal baselines by leveraging AraBERT's morphological priors to resolve ambiguities that visual features alone cannot disambiguate. Contributions include: (1) an open-source multimodal Arabic OCR/HWR framework; (2) empirical demonstration of improved CER/WER over state-of-the-art unimodal systems; and (3) comprehensive analysis of cross-modal fusion strategies for Arabic script recognition.