Empirical Evaluation of Language Models for Arabic Financial Sentiment Analysis: A Comparative Study of Pretrained and General Large Language Models

Abstract

This study presents a comparative study on the effectiveness of large and pre-trained language models in analyzing sentiment in Arabic financial news. While sentiment analysis using AI has advanced significantly in English-language finance, similar progress in Arabic remains limited. Most existing Arabic models, such as AraBERT and CAMeLBERT, are trained on general or social media posts and struggle with domain-specific content like financial data. At the same time, modern LLMs like ChatGPT and Gemini have not been thoroughly tested on Arabic financial data. Our work addresses these gaps by evaluating five models—ChatGPT, Gemini, Nuha (an Arabic LLM), AraBERT, and CAMeLBERT—on a dataset of 10,166 financial news headlines from Argaam, a leading Saudi financial platform. We conduct three key experiments. First, we conduct a standard sentiment classification comparison across all models. Second, prompt engineering experiment using zero-shot, one-shot, and three-shot strategies. Finally, persona-driven analysis where models respond from the viewpoint of different financial roles. The results show that Nuha consistently delivers the highest performance, except when prompts include user roles. They show promising results, but did not achieve the highest performance. ChatGPT maintains stable results across prompt variations and shows the most remarkable results when there are persona-driven prompts. Gemini performs well in zero-shot but degrades with added context. PLMs underperform, highlighting the need for better domain adaptation. Overall, this study provides practical insights into model behavior under different prompting conditions, emphasizes the importance of cultural alignment, and introduces persona-driven prompting as a valuable strategy for Arabic financial sentiment analysis.