This work introduces PERCEPT, the first publicly available large-scale Persian-English code-mixed corpus annotated with Universal Dependencies POS tags for code-mixed words, and conducts the first comprehensive linguistic analysis of Persian-English code-mixing across multiple social media platforms.
Abstract
Social media has become a major venue for multilingual communication, where users frequently mix multiple languages within a single utterance. Although code-mixed corpora have been developed for several language pairs, Persian-English code-mixing remains relatively underexplored. Existing Persian resources lack Universal Dependencies (UD) part-of-speech (POS) annotations for code-mixed words, limiting both linguistic analyses and the development of syntax-aware NLP models. To address this gap, we introduce PERCEPT, the first publicly available large-scale Persian-English code-mixed corpus annotated with Universal Dependencies POS tags for code-mixed words. The dataset comprises 6,800 posts collected from X, Instagram, and Digikala. We further present an LLM-assisted annotation framework that automatically assigns POS tags and document-level topics. Human evaluation demonstrates high agreement between the automatically generated annotations and gold annotations, confirming the reliability of the annotations. Using PERCEPT, we conduct the first comprehensive linguistic analysis of Persian-English code-mixing across multiple social media platforms. Our analyses reveal that nouns are the predominant category for code-mixed words, while the distributions of other POS categories vary across platforms. We further find that the positional distribution of code-mixed words is remarkably consistent across platforms, whereas the triggering effect is substantially more pronounced in Digikala. PERCEPT is publicly available at https://github.com/kalhorghazal/PERCEPT.
Part Of Speech (POS) tagging is a fundamental task in Natural Language Processing (NLP) that assigns grammatical labels to words in a sentence. Code mixed text, which entails switching between two or more languages within a single conversation or a sentence, presents challenges for POS tagging. This investigation entailed a comprehensive study of deep learning approaches for cross linguistic POS tagging, focused on English Malayalam code mixed data prevalent on social media platforms. The study was carried out on linguistically complex and varied English Malayalam code mixed text from social media platforms with informal spellings, language switching, transliteration, slang, and unclear grammatical boundaries, reflecting the characteristics of informal online communication. We provide the first structured analysis of micro macro divergence and rare class behavior in English Malayalam code mixed POS tagging.
We evaluated 14 state of the art model configurations that span traditional sequence labeling approaches and multilingual transformer architectures. Models were compared using standard performance metrics prevalent in the domain of data science, supplemented by normalized confusion matrices, error prone tag identification and micro-macro F1 gap analysis.
Our results showed that CRF (No Lang) emerged as the most balanced model overall on macro F1 (all classes) of 0.8170, while (BiLSTM + CRF) achieved the highest macro F1 (seen classes) of 0.8831, precision of 0.9167, and recall of 0.875, though this reflects strong performance concentrated on frequent tag classes rather than balanced coverage across the full tag set. Notably, the pretrained multilingual transformers (mBERT, MuRIL), despite prior exposure to Malayalam during pretraining, were outperformed on several key metrics by CRF and BiLSTM models trained directly on the code mixed dataset.
This finding was contrary to our expectation that existing multilingual knowledge would translate into a clear advantage on this task and merits further investigation.
Unknown authors· Frontiers in Big Data· 0 citations
Large Language Models (LLMs) have transformed conversational AI, yet high-quality multilingual code-mixed dialogue resources remain scarce, particularly for Indic languages where speakers naturally alternate between English and their native language in both native-script and Romanized forms. We present IndicTalk, one of the largest multilingual Indic code-mixed conversational corpora, comprising over 13,28,604 event-grounded multi-turn conversations across 18 language varieties covering 9 Indic languages. The corpus is generated through a fully automated pipeline that combines real-world news grounding, persona-conditioned dialogue generation using multilingual LLMs, and automatic quality validation. Extensive linguistic, automatic, and human evaluations demonstrate that IndicTalk produces fluent, coherent, and naturally code-mixed conversations across both script variants. We will release IndicTalk to support the development and evaluation of multilingual conversational AI for underrepresented Indic languages. The dataset is available at: https://huggingface.co/datasets/LingoIITGN/IndicTalk .
Part-of-speech (POS) tagging for low-resource languages is limited more by the expense of generating labeled training data than by algorithmic constraints. This study evaluates the extent to which a length-stratified, minimally-supervised pipeline can recover tagging accuracy for the Assamese language using a labeled seed of only 3 to 5 sentences per distinct sentence length (142 to 228 sentences in total), compared to a weak-supervision classifier trained on a conventional 80% corpus split (2,840 sentences). An evidence-gated cascade is constructed, comprising Brown-style word clustering, a seed-ambiguity audit with context-based disambiguation, confidence-gated cluster labeling, and empirically validated rule fallbacks. This approach achieves 94.9% to 95.0% accuracy but only 19.8% to 20.0% token coverage from its minimal seed. To address the coverage gap, every remaining token is backfilled using a classifier trained on the pipeline’s accumulated evidence, resulting in 100% coverage at 84.7% to 86.0% accuracy (depending on seed size), as evaluated against a fixed, shared test set for direct comparison with the larger-budget baseline (90.8% accuracy, same coverage, same test set). Three negative results are reported alongside the positive findings: Brown-style clustering provides negligible benefit at this scale, Viterbi sequence decoding does not transfer to the backfill classifier from the larger-budget setting, and naive self-training fails due to confirmation bias. Additionally, through five replicated random seed draws per condition, it is demonstrated that the residual seed-size effect (3 versus 5 sentences per length, closing approximately 21% of the accuracy gap to the larger-budget baseline) is statistically significant rather than the result of a single favorable draw.
B. Sarma, Rupam Baruah, Diganta Baishya· International journal of com...· 0 citations
Recently, contact linguistics has become increasingly interested in multiword units. At the same time, the code-copying framework (CCF) includes the notion of mixed copies (MCs) that are in-between global copies (‘borrowing’) and selective copies (‘structural change’) and illustrate the transition between the lexicon and grammar. The research question is: What types of MCs occur in English-Estonian bilingual speech?
The data were transcribed, and MCs identified, annotated, and classified according to their structure. English items were searched for in Estonian dictionaries to establish their Estonian equivalents or conventionalization of such items. The frequencies of MCs and their Estonian equivalents were also searched on Google to determine whether the MCs occur outside the corpus.
Three datasets were analysed: written texts from 44 blogs (385,124 tokens), spoken data from 10 vlogs (117,555 tokens), and 8 podcasts (77,277 tokens). Quantitative analyses of the various MC types were conducted, followed by a qualitative analysis of representative examples.
Compound nouns constitute the majority of MCs, followed by idioms, phrasal compounds, and a small number of compound verbs. No frame-changing MCs (i.e., MCs resulting in grammatical change) were attested. Since compound nouns and analytic verbs occur in both languages, structural similarity may be a facilitating factor in copying.
The notion of MCs is not widely used. Research typically focuses on particular types of items (e.g., compound nouns or verbs); here, however, the question is reversed: which types of items yield MCs?
It was established that the proportion of MCs in the data is comparable to that of selective copies. Within MCs, the globally copied element renders the remaining part more specific, highlighting the importance of meaning in contact-induced language change. MCs are also present on the Estonian internet and, in some cases, outnumber their Estonian equivalents, if such equivalents exist.
A. Verschik, H. Kask· International Journal of Bil...· 0 citations
Persian (Farsi) is often described as a low-resource language in natural language processing, but that label collapses distinct shortages into a single category. This paper argues that Persian is more precisely described as annotation-scarce, provided that the term is understood as a property of its NLP resource ecology rather than an intrinsic property of the language. The review covers 34 representative Persian text resources available by July 2026 and adds three quantitative cross-checks. First, independent web measurements place Persian among roughly the twenty most visible content languages: W3Techs reports Persian on about 0.9% of websites with a known content language, while Common Crawl CC-MAIN-2026-30 identifies Persian as the primary language of 0.7039% of HTML pages. Second, a selective speech review shows a long resource trajectory from FARSDAT to recent corpora containing hundreds or thousands of hours of speech. Third, a matched Persian-English comparison normalizes task-specific annotation volumes by relative Common Crawl web presence. The resulting ratios vary sharply: Persian syntax and news NER are comparatively dense, whereas natural-language inference falls below the web-proportional baseline. The evidence therefore does not support a simple claim that Persian is globally deficient in labeled volume. Instead, annotation scarcity is expressed through uneven task and domain coverage, incompatible schemes, access and documentation friction, and limited supervision for specialist domains, preference data, and varieties beyond standard Iranian Persian.
MohammadHossein Mortazavi, Mostafa Salehi, Hadi Veisi· 0 citations
This work implements a full neural annotation pipeline (morpheme segmentation, POS tagging, glossing) for Irabu Ryukyuan using deliberately small, transparent BiLSTM-CRF models, and concludes with a concrete recommendation for documentation practice: annotate quadrilinearly - text, POS, gloss, translation.
