TL;DR
An in-depth evaluation of instruction tuning for Arabic NLP tasks using three prominent LLMs: LLaMA 3.1-8B, AceGPT-v2-8B, and Qwen3-8B shows that instruction tuning consistently improves performance across most tasks, with notable variations in effectiveness across different tasks and prompts.
A bidirectional Mamba encoder pretrained via masked language modeling on a corpus combining Arabic Wikipedia and CulturaX text is introduced, trained end-to-end on four consumer-grade NVIDIA RTX 2080Ti GPUs (11GB) over approximately ten days.
: Arabic-English code-switching is common in online communication, where users mix both languages within the same sentence. Mixed scripts, dialectal Arabic, Arabizi, and inconsistent spelling create instability for NLP models and make tasks such as topic classification difficult. Previous work has typically focused either on improving transformer architectures through ensembles and multi view modeling, or on enhancing inference through prompt design in large language models. However, these directions are rarely evaluated under the same experimental setup. In this paper, we provide a controlled comparison between architecture based optimization and prompting based inference for Arabic-English code-switched topic classification, and we release ArEnTC , a 105k-sentence Arabic-English code-switched dataset annotated for nine topics. Architectural experiments include single model baselines, voting ensembles, stacked meta learning, and translation based multi view inference. Prompting experiments evaluate zero shot, few shot, retrieval augmented generation (RAG), and reasoning based strategies using LLaMA 3.3 70B without fine tuning. While architectural extensions progressively improve performance, reaching 0.92 Macro F1 with translation based multi view and Random Forest, retrieval augmented few shot prompting achieves 0.98 Macro F1, surpassing all ensemble configurations. The results indicate that inference time conditioning can be more effective than increasing architectural complexity for this task.
Jais 2 is a family of Arabic-centric large language models developed jointly by MBZUAI, Cerebras, and Inception, designed to advance Arabic-centric language modeling, with strong performance across the Arabic and culturally grounded benchmarks evaluated in this report. The family includes, to our knowledge, the largest open Arabic-centric LLM trained from scratch at 70B parameters, and a competitive 8B-parameter variant among the evaluated open models. A custom Arabic-centric vocabulary enables efficient training and inference. In addition, an optimized architecture and training recipe yield highly compute-efficient training. With a substantially smaller token budget than comparable models, Jais 2 achieves strong Arabic performance on the benchmarks considered in this report and competitive English results. The models obtain leading results among the evaluated open models on OALL2 and AraGen. They also perform strongly on several culturally grounded Arabic benchmarks, including poetry, religion, cuisine, and dream interpretation, as well as in general tasks such as translation and summarization. We release the models in HuggingFace under a commercially permissive license. Jais 2 70B is also released as a chat app on the Web, iOS, and Android; it runs on Cerebras hardware, delivering up to 2,000 tokens per second, and enabling high-throughput Arabic-centric chat serving in our deployment setting. By uniting scale, linguistic diversity, cultural fidelity, openness, and speed, Jais 2 provides an open-weight foundation intended to support further research and development in Arabic-centric LLMs.
Retrieval-Augmented Generation (RAG) has become a standard technique to ground large language model outputs in external knowledge. However, evaluating RAG systems for Arabic remains problematic because traditional lexical metrics such as ROUGE and BLEU were designed for English, a language with limited morphological variation. Arabic's rich derivational system where a single triliteral root can produce dozens of surface forms causes these metrics to penalize correct paraphrasing while being completely bling to factual hallucination. This paper introduces a holistic evaluation framework that combines hybrid retrieval (BM25 with dense embeddings from BGE-M3) and a multi-dimensional scoring suite. We compare six LLMs three Arabic specialized (ALLaM-7B, Fanar-1-9B, Noon-7b) and three multilingual (Llama-3.3-70B, Qwen-2.5-7B, BLOOM-7B) across 300 queries over a general Arabic corpus of 30 documents. Our hybrid retriever achieves aRecall@5 of 0.942 and an MRR of0.883. The central empirical finding is a metric mirage: the Pearson correlation between ROUGE-1 and semantic similarity (measured by BGE-M3) is only 0.317, meaning that more than 90% of factual variance is invisible to lexical overlap metrics. Llama-3.3-70B achieves the highest composite score (2.703/4), driven by superior named entity recognition accuracy (0.463). However, the smaller Arabic-specialized ALLaM-7B exhibits strong performance across several of evaluation metrics—particularly named entity recognition (NER), ROUGE-1, ROUGE-2, and BLEU. https://github.com/AAA20121/Beyond-Lexical-Overlap.git.
Kazakh automatic speech recognition (ASR) presents a persistent challenge for large-scale multilingual models. This paper presents a systematic evaluation of 27 ASR pipeline configurations (three ASR models × three VAD methods × three post-processing strategies) on the Kazakh Speech Dataset (KSD), examining the contribution of model fine-tuning, voice activity detection (VAD) preprocessing, and large language model (LLM) post-correction and benchmarking the resulting pipelines against two non-Whisper foundation models. Language-specific fine-tuning reduces Word Error Rate (WER) from 43.20% (generic Whisper-large-v3) to 11.88% (Kazakh fine-tuned Whisper-turbo), a 31.32-percentage-point absolute reduction (72.5% relative; p < 0.001, bootstrap test); the effect persists after controlling for model size (generic Whisper-large-v3-turbo, 18.92%, vs. the same architecture after fine-tuning, 11.88%; p < 0.001). VAD preprocessing consistently degrades performance. Zero-shot post-correction with general-purpose LLMs yields no benefit and adds substantial latency: Gemma-2-9B and Qwen2.5-7B raise WER by 5.5 and 7.2 percentage points at real-time factors of 0.52 and 0.30, and a larger 32B model still degrades accuracy (+10.8 points), indicating that scale is not the limiting factor. Among all systems evaluated, a larger multilingual foundation model, SeamlessM4T-v2 (9.72% WER), outperforms the fine-tuned Whisper, showing that for Kazakh model coverage matters more than pipeline engineering. Character-level error analysis identifies systematic confusion between Kazakh-specific and Russian Cyrillic characters as a dominant error source. These findings establish that, for Kazakh under the evaluated conditions, model choice dominates pipeline add-ons: fine-tuning is essential, VAD and zero-shot LLM correction consistently hurt, and a strong multilingual model sets the best result; we further discuss the extent to which these conclusions extend to typologically similar Kipchak-Turkic languages.