Sep 2026· Conference on Empirical Methods in Natural Language Processing· Vol abs/2609.19916· 0 citations· 43 references
Computer Science
TL;DR
KoNeoBench is a benchmark for evaluating LLMs' understanding of Korean neologisms, built on 1,785 Korean neologisms attested in online news since 2020 and curated through expert lexicographic review.
Abstract
Large language models (LLMs) are typically evaluated on static benchmarks, even though natural language constantly evolves through newly emerging words and meanings. Existing Korean benchmarks are centered on established vocabulary and therefore provide limited coverage of such recent lexical change, and their English-oriented design makes it difficult to assess the typological properties of Korean, in which content words combine productively with functional morphemes. In this paper, we introduce KoNeoBench, a benchmark for evaluating LLMs'understanding of Korean neologisms. KoNeoBench is built on 1,785 Korean neologisms attested in online news since 2020 and curated through expert lexicographic review. Each entry provides usage examples, word-formation analyses, and dictionary-style definitions. Based on this resource, we define four tasks and report results on recent models, together with a human baseline. Our experiments show that current LLMs exhibit clear limitations in recovering source components, distinguishing semantic categories, and generating accurate definitions. These results reveal specific aspects of recent Korean lexical change that remain challenging for current LLMs. KoNeoBench is available at https://github.com/bcmilab/ko-neobench/ .
CNeo-Bench, a benchmark of 4,759 Chinese neologisms with reference definitions, is introduced, organized into five top-level categories and nine subcategories by the linguistic mechanism behind each expression, paired with a two-tier evaluation framework that separates whether a model can describe a neologism from whet...
Kai-Yan Zhao, Zhong-Tao Miao, Zhe-Yong Xie et al.· 0 citations
A comprehensive evaluation of recent large language models (LLMs) across three idiom-related tasks: paraphrasing, idiom span detection, and meaning identification, leveraging zero-shot and few-shot prompting strategies reveals substantial variability in model performance.
Mousumi Akter, Md. Faiyaz Abdullah Sayeedi, Nurul Labib Sayeedi et al.· arXiv.org· 0 citations
It is suggested that LLMs can serve as auxiliary annotators for sensitive-language detection in historical materials, provided that human oversight and contextual interpretation remain central to annotation workflows.
Ya-Hui Zhao, Clemencia Siro, L. Hollink et al.· 0 citations
E-CONAN diagnostics dataset is introduced, a freely available dataset manually-annotated with coarse-grained and fine-grained categories based on the authors' proposed Arabic hierarchy, which helps NLU designers better understand their models by doing error analysis and in-depth investigation.
Khloud Al Jallad, Nada Ghneim, Ghaida Rebdawi· arXiv.org· 0 citations
This paper proposes automated methods to construct high-quality WordNets using large language models (LLMs) to generate missing lemmas to address the synset shortfall in non-English and low-resource languages.
Johann Bergh, J. Waitelonis, Melanie Siegel· 0 citations
AraDynFact is introduced, a novel dynamic evaluation framework designed to rigorously assess the factual Arabic knowledge embedded in LLMs, and employs a dynamic approach to extract factual information and generate rich and answerable questions in a fast and automatic way.
Ignacio Iacobacci, Faroq Al-Tam, Zhao-Zhi Qian et al.· Proceedings of the 2026 Conf...· 0 citations