Jun 2026· International Journal of Computer Science and Engineering· Vol 14, pp. 34-41· 0 citations
TL;DR
This work focuses on word-level Language Identification (LID) for transliterated text in informal Roman transliteration, and relies on character-based TF–IDF features and a set of traditional machine-learning models.
Abstract
In recent years, social media users in multilingual regions have begun mixing languages more freely, and Telugu–English combinations are among the most common examples in India. Much of this content appears in informal Roman transliteration, and the lack of uniform spelling makes automatic processing difficult. In this work, we focus on word-level Language Identification (LID) for such transliterated text. Our approach relies on character-based TF–IDF features and a set of traditional machine-learning models. In this study, we worked with four different models—Multinomial Naive Bayes, Logistic Regression, Random Forest, and Support Vector Machine—and evaluated them on an annotated set that included Telugu, English, Named Entity, and Universal tokens. Among the four, the SVM turned out to be the strongest, reaching an accuracy of 86% along with an F1-score of 0.85. The study also brings out some practical issues with real-world transliterated text, particularly class imbalance and the wide range of spelling variations. We conclude with possible directions for improvement, including the use of neural and transformer-based models that might capture more contextual cues in future versions of this system.
Overall, this work demonstrates that incorporating explicit linguistic information, including language identity, sentiment polarity, and intensifier information, improves sentiment classification of Gujarati–English code-mixed text.
Chirag D. Shah, Shailesh A. Chaudhari· International journal of com...· 0 citations
The first machine translation study for the English and Pnar language pair is presented, establishing the first quantitative benchmark for this language pair and discussing future directions toward neural and multilingual machine translation for Pnar.
This study explores a low-resource approach to detecting hate speech in English and Swahili code-switched text by fine-tuning pre-trained language models, and shows that fine-tuning modern language models can offer a practical and scalable solution for hate speech detection in multilingual environments.
Kipkebut Andrew, Jepkemei Betty· International Journal for Re...· 0 citations
The findings underscore the potential of advanced NLP techniques to overcome language-specific challenges, providing a foundation for future research in multilingual plagiarism detection and enhancing the development of tools for other languages facing similar challenges.
Hanan Fawzy, Ahmad Salah, Heba El-Fiqi et al.· Informatica· 0 citations
Social scientists often machine-translate foreign-language texts into English and apply English-based natural language processing tools without systematically evaluating translation quality or annotation efficiency. To address this problem, this study provides evidence-based guidance for researchers applying English-centric natural language processing to domain-specific foreign-language corpora. We provide and empirically validate a structured framework combining multi-system machine translation evaluation and active learning for domain-specific text classification. Using 11,493 parallel Spanish and Arabic sentences aligned to English, we compare four machine translation systems (Google Translate, Deep, DeepL, OPUS) using SacreBLEU, METEOR, COMET, and BERTScore quality scores. Across languages and metrics, machine translation systems yield statistically comparable performance. We then evaluate eight active learning strategies using ConfliBERT for political conflict classification under a 20% annotation budget, corresponding to 1,155 samples from the training split. Binary classification exceeds F1 = 0.90, while QuadClass multi-class performance peaks around F $1~\approx ~0.75$ . The Ensemble Intersection strategy achieves the highest performance in 53% of tasks and often matches or surpasses full-dataset results using only a fraction of labeled data. These results provide a practical workflow for researchers using English-based natural language processing tools on foreign-language, domain-specific corpora.
Naif Alatrush, Luay Abdeljaber, Javier Osorio et al.· IEEE Access· 0 citations
Large language models (LLMs) process text through subword tokenizers rather than directly reading characters or words. Because these tokenizers are trained predominantly on English-centric corpora, they introduce a systematic and often overlooked disadvantage for many non-English languages. In this work, we quantify this tokenizer tax for Indian languages using the FLORES-200 parallel corpus, measuring tokenization fertility across six widely used tokenizers and fourteen languages. Under cl100k_base (used by GPT-3.5 and GPT-4), Indian languages experience an average 8.0x tokenization tax relative to English, reaching 13.0x for Malayalam, reducing the effective context window to as little as 12% of that available to English users for equivalent semantic content. We identify the primary mechanism behind this disparity: failed byte-pair merges that leave text fragmented into single-byte tokens, with merge failure strongly correlating with tokenizer tax (Pearson r = 0.89). We further show that this phenomenon is not an inherent property of Indic scripts but a consequence of tokenizer design. Multilingual tokenizers such as XLM-R and OpenAI's o200k_base reduce the average Indic tokenizer tax by 73%, demonstrating that the disparity is largely remediable. Beyond token statistics, we quantify a practical consequence by showing that, under fixed context budgets, Indian-language documents preserve substantially less original content than equivalent English documents. Finally, we examine the relationship between tokenizer fertility and reading comprehension performance on the Belebele benchmark, finding that the apparent correlation is largely explained by language resource availability rather than tokenizer behavior alone.