Evaluating Small Open-Weight LLMs on Code-Mixed Nepali-English Named Entity Recognition

Abstract

This paper presents an evaluation of small open-weight large language models (LLMs) on named entity recognition (NER) for code-mixed Nepali-English text. No dedicated Nepali-English NER resource currently exists, to the best of the authors’ knowledge. The LinCE benchmark [Aguilar et al. 2020] includes Nepali-English as one of its four language pairs, but only for the language identification task; its NER coverage is limited to Spanish-English, Hindi-English, and Modern Standard Arabic-Egyptian Arabic. A 274-sentence gold-annotated dataset of code-mixed Nepali-English text is constructed from YouTube comments. A blind re-annotation of 45 sentences, 16.4% of the dataset, by the same annotator gives a Cohen’s kappa of 0.918, with 98.5% raw agreement. Four small open-weight LLMs are evaluated under identical one-shot prompting and greedy decoding conditions: Llama-3.2-1B, Qwen2.5-1.5B, Gemma-2-2B, and Phi-3.5-mini. Three complementary metrics are used: entity-level F1, a schema adherence rate that measures structural output-format compliance, and F1 stratified by code-mixing density. All four models perform poorly in absolute terms, with F1 no higher than 0.12. Schema adherence and entity-level correctness turn out to be largely independent of one another; the model with the highest F1 point estimate, Phi-3.5-mini, has close to the lowest adherence score of the four, though bootstrap confidence intervals show Phi-3.5-mini, Gemma-2-2B, and Qwen2.5- 1.5B are not statistically distinguishable from one another on F1 alone. A supervised baseline (XLM-RoBERTa base, fine-tuned under 5-fold cross-validation on the same data) reaches an F1 of 0.422, roughly 3.6 times higher than the best-performing LLM, indicating that the task is learnable from this dataset and that the LLMs’ weak performance reflects a genuine capability gap rather than an unlearnable or anomalous dataset. Organization entities are the hardest category for every model, including the supervised baseline, mainly because of errors at multi-token boundaries. The dataset, annotation guidelines, and evaluation pipeline are made available to support future work on Nepali-English code-mixed NLP.