Open access
Bulgarian Massive Multitask Language Understanding Benchmark
S. Koeva
I. Stoyanova
Dimiter Georgiev
S. Leseva
Valentina Stefanova
Maria Andreeva Todorova
Tsvetana Ivanova Dimitrova
Hristina Kukova
Mihaela Moskova
Tinko Tinchev
2026 · International Conference on Language Resources and Evaluation · pp. 4658-4672 · 1 citation
· 38 references
Computer Science
Abstract
Assessing the broad general knowledge of Large Language Models (LLMs) across multiple domains in Bulgarian remains challenging due to the limited availability of Bulgarian evaluation benchmarks. To address this gap, we introduce the Bulgarian Massive Multitask Language Understanding benchmark ( MMLU-BG ), designed to evaluate whether LLMs possess generalised knowledge capabilities beyond simple text prediction in Bulgarian. This paper presents the structure, the development protocol