Skip to content

Beyond Cultural Knowledge: Evaluating Arabic Cultural Appropriateness of Large Language Models

Jul 2026 · 0 citations · 29 references
Computer Science

TL;DR

A AraBehave is introduced: 1,623 culturally grounded, open-ended Arabic prompts with 29,214 cultural-appropriateness judgments from native speakers across several Arab regions, plus a scoring model whose predictions correlate strongly with human judgments on unseen systems.

Abstract

Large language models (LLMs) increasingly serve users whose expectations are shaped by their cultural context, yet most cultural evaluations test what a model knows rather than how it behaves when giving open-ended recommendations, opinions, and guidance. We introduce AraBehave: 1,623 culturally grounded, open-ended Arabic prompts with 29,214 cultural-appropriateness judgments from native speakers across several Arab regions, plus a scoring model whose predictions correlate strongly with human judgments on unseen systems (Pearson r=0.74). Evaluating three Arabic-centric and three frontier LLMs, we find that cultural appropriateness is not a single capability but decomposes into two largely independent components: normative stance and grounded cultural accuracy. The best general-purpose and best Arabic-centric models score identically (3.84 vs. 3.83 of 5) yet almost never fail for the same reason: general-purpose models exhibit strong factual grounding but a culturally inappropriate normative stance, being penalized for secular framing and false balance on culturally settled matters (28--33% of their low-score rationales), while the best Arabic-centric model adopts the expected stance but is penalized for fabricated hadith and misquoted verses (29%). Stance is cheap and fragile: one sentence of cultural instruction lifts Gemini to 4.57, above every Arabic-specialized model. Conversely, a generic ``answer clearly and objectively''prompt costs Allam-7B 0.68 points, while asking the same questions in English lowers scores for every model but one. Grounding instead tracks scale and Arabic alignment data, and disappears when culturally aware instruction tuning is replaced by a culture-neutral corpus. General safety benchmarks see none of this: they saturate above 89 while cultural scores span 2.71-3.84. We will release the benchmark, annotations, and the scoring model.

View source

Similar papers

Open access Sep 2026

Toward a Culturally Embedded Arabic LLM: Localizing Linguistic AI for the Arab World

Arabic large language models (LLMs) increasingly support communication, education, public services, and knowledge access, yet fluent Arabic generation does not necessarily constitute culturally appropriate understanding. This paper examines the problem of cultural and pragmatic misalignment in contemporary Arabic-capab...

Fahd A. Alhathloul, N. Alowedi · 0 citations
#artificial intelligence Preprint Aug 2026

Beyond Fluency: A Rubric-Based Benchmark for Evaluating Saudi Dialect and Cultural Competence in Large Language Models

Large language models are increasingly deployed in Arabic-speaking markets, yet standard benchmarks overwhelmingly reward Modern Standard Arabic (MSA) fluency while leaving dialectal and culturally grounded competence unmeasured. This gap is consequential: everyday Arabic is largely dialectal, and dialect encodes socia...

Ghassan Al-Sumaidaee, Sajjad Abdoli, Ahmed Rashad et al. · 0 citations
Review Sep 2026

Large language models underestimate and partly misrepresent cultural variation in everyday norms

A key aspect of culture is a society's norms about everyday behavior. How accurately do large language models (LLMs) represent cultural differences in such norms? To answer this question we used the recent Global Study of Everyday Norms (GSEN), which collected ratings of 150 scenarios in 90 societies, as the human benc...

Kimmo Eriksson, I. Vartanova, P. Strimling · 0 citations
Open access 2026

CURE: Cultural Value-based Reasoning for Enhancing the Cultural Adaptability of Large Language Models

While large language models (LLMs) are widely used across cultures, they often generate culturally inappropriate responses in unfamiliar cultural contexts due to biases embedded in their training data. Existing approaches primarily rely on expanding static cultural knowledge, which fails to capture the inherently rel...

Mirae Han, Harksoo Kim · 0 citations
#artificial intelligence Preprint Sep 2026

VakyArth: Evaluating Pragmatic Competence in LLMs across Indic Languages

Real-world communication often requires pragmatic reasoning: interpreting meanings implied through context and cultural convention rather than stated literally. Existing pragmatic evaluation remains largely limited to English and high-resource languages, leaving Indic languages unexplored despite their linguistic and c...

Usneek Singh, Poorvaja Veera Balaji Kumar, Parth Nanda et al. · 0 citations
Open access 2026

Beyond Literal Meaning: How LLMs Interpret Yemeni Proverbs

Results show that instruction-tuned models like GPT-4o and Gemini 1.5 Pro outperform smaller models in both automatic and human evaluations, and LLM-as-a-Judge evaluation correlates strongly with human assessment.

Nasser Thmer, Ali Allaith, Muhammad Shoaib · 0 citations

Related blog posts

MIT News · Artificial Intelligence Nov 17, 2026

Learning the language of machine learning

By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.