NerAxom: a BIO-tagged NER dataset and hybrid neural-rule framework for Assamese

Abstract

Named entity recognition (NER) in low-resource, morphologically rich languages such as Assamese (ISO 639-3: asm) remains a significant challenge due to the scarcity of annotated corpora and the limited applicability of models designed for resource-rich languages. Existing Assamese NER resources suffer from critical limitations: WikiAnn provides broad language coverage but insufficient data volume for neural model training; AsNER, while a gold-standard corpus, supports only five entity categories and lacks a formal tagging scheme, restricting its utility for downstream tasks such as relation extraction and information retrieval. Furthermore, prior Assamese NER systems have relied predominantly on traditional tagging approaches and classical machine learning methods, with limited exploration of modern pre-trained language models and linguistically motivated post-processing strategies. To address these gaps, we present NerAxom, a BIO-tagged NER dataset for Assamese comprising 4,173 sentences and 106,046 tokens annotated across seven entity categories: Person (PER), Location (LOC), Organization (ORG), Date (DATE), Work_of_Art (WOA), Occupation (OCC), and Number (NUM). The dataset was independently annotated by two trained native speakers, achieving an inter-annotator agreement of κ=0.82 (Cohen’s Kappa), with disagreements resolved through expert linguist adjudication. We evaluate NerAxom using two modeling paradigms: (i) a BiLSTM–CRF model with an attention mechanism, tested with FastText, BERT, and MuRIL embeddings; and (ii) direct fine-tuning of the MuRIL transformer as a token classifier. Among embedding-based models, MuRIL yields the highest F1-score of 68%, outperforming FastText (62%) and BERT (64%). Fine-tuning MuRIL directly as a token classifier achieves an F1-score of 70%, establishing a competitive transformer baseline. To address entity misclassifications arising from Assamese morphological complexity, we further introduce a set of language-specific post-processing rules based on morphological suffixes and keyword cues. The hybrid system combining MuRIL embeddings in the BiLSTM–CRF+Attention architecture with these linguistic rules achieves an F1-score of 71% and an overall accuracy of 82% on Assamese Wikipedia biographical text, competitive with the fine-tuned MuRIL transformer (70% F1) and demonstrating that linguistically informed post-processing provides complementary gains over embedding-based neural baselines. The rule component yields the largest category-wise gains for LOC (+20 F1), WOA (+11 F1), and ORG (+9 F1). The NerAxom dataset is publicly available to support further NER research in Assamese and related low-resource Indic languages.