Skip to content

KoNeoBench: A Curated Evaluation Dataset for LLM Understanding of Korean Neologisms

Sep 2026 · Conference on Empirical Methods in Natural Language Processing · Vol abs/2609.19916 · 0 citations · 43 references
Computer Science

TL;DR

KoNeoBench is a benchmark for evaluating LLMs' understanding of Korean neologisms, built on 1,785 Korean neologisms attested in online news since 2020 and curated through expert lexicographic review.

Abstract

Large language models (LLMs) are typically evaluated on static benchmarks, even though natural language constantly evolves through newly emerging words and meanings. Existing Korean benchmarks are centered on established vocabulary and therefore provide limited coverage of such recent lexical change, and their English-oriented design makes it difficult to assess the typological properties of Korean, in which content words combine productively with functional morphemes. In this paper, we introduce KoNeoBench, a benchmark for evaluating LLMs'understanding of Korean neologisms. KoNeoBench is built on 1,785 Korean neologisms attested in online news since 2020 and curated through expert lexicographic review. Each entry provides usage examples, word-formation analyses, and dictionary-style definitions. Based on this resource, we define four tasks and report results on recent models, together with a human baseline. Our experiments show that current LLMs exhibit clear limitations in recovering source components, distinguishing semantic categories, and generating accurate definitions. These results reveal specific aspects of recent Korean lexical change that remain challenging for current LLMs. KoNeoBench is available at https://github.com/bcmilab/ko-neobench/ .

Read PDF

Similar papers

#natural language process... Preprint Aug 2026

CNeo-Bench: Diagnosing Large Language Models on Chinese Neologisms

CNeo-Bench, a benchmark of 4,759 Chinese neologisms with reference definitions, is introduced, organized into five top-level categories and nine subcategories by the linguistic mechanism behind each expression, paired with a two-tier evaluation framework that separates whether a model can describe a neologism from whet...

Kai-Yan Zhao, Zhong-Tao Miao, Zhe-Yong Xie et al. · 0 citations

To What Extent Do Large Language Models Understand Bangla Idioms?

A comprehensive evaluation of recent large language models (LLMs) across three idiom-related tasks: paraphrasing, idiom span detection, and meaning identification, leveraging zero-shot and few-shot prompting strategies reveals substantial variability in model performance.

Mousumi Akter, Md. Faiyaz Abdullah Sayeedi, Nurul Labib Sayeedi et al. · 0 citations

E-CONAN (Entailment, CONtradition And Neutral) Diagnostics Dataset Investigating Linguistic Phenomena in Arabic Natural Language Understanding

E-CONAN diagnostics dataset is introduced, a freely available dataset manually-annotated with coarse-grained and fine-grained categories based on the authors' proposed Arabic hierarchy, which helps NLU designers better understand their models by doing error analysis and in-depth investigation.

Khloud Al Jallad, Nada Ghneim, Ghaida Rebdawi · 0 citations

Leveraging LLMs to Automatically Construct WordNets as Bilingual Resources

This paper proposes automated methods to construct high-quality WordNets using large language models (LLMs) to generate missing lemmas to address the synset shortfall in non-English and low-resource languages.

Johann Bergh, J. Waitelonis, Melanie Siegel · 0 citations
#natural language process... Conference Open access Sep 2026

AraDynFact: Dynamic Evaluation of Factual Knowledge in Arabic

AraDynFact is introduced, a novel dynamic evaluation framework designed to rigorously assess the factual Arabic knowledge embedded in LLMs, and employs a dynamic approach to extract factual information and generate rich and answerable questions in a fast and automatic way.

Ignacio Iacobacci, Faroq Al-Tam, Zhao-Zhi Qian et al. · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.