We present HelaBERT, a family of two BERT-based masked language models pre-trained from scratch on approximately 1 billion tokens of Sinhala text sourced from MADLAD-400, CulturaX, and a custom corpus comprising news articles, Sinhala Wikipedia, and web crawl data. HelaBERT-Small (~23.3M parameters, 6 layers) and HelaBERT-Large (~110M parameters, 12 layers) both use a SentencePiece Unigram tokenizer (vocabulary size 32,000) tailored to Sinhala's agglutinative morphology and complex script. We evaluate both models on four downstream Sinhala text classification tasks: news category classification, news source classification, sentiment analysis, and writing style classification, using 5 independent seed runs with stratified 80/20 train/test splits. We additionally propose a dual pooling classification head and evaluate it systematically across all four tasks, finding consistent improvements on sentiment analysis and a moderate gain on news category classification for HelaBERT-Small, while the standard [CLS]-linear head remains competitive on news source classification, a headline-level task with short average input length. We release both models to support further research in Sinhala NLP.
This study successfully proposes a Long Short-Term Memory (LSTM)-based model for automatic classification of Indonesian regional song lyrics by language, demonstrating that LSTM effectively captures sequential linguistic patterns and contextual relationships within regional languages.
Muhammad Rizky, Anandita Priatama, Aviv Yuniar Rahman et al.· Buana Information Technology...· 0 citations
India is home to an incredible number of languages which leads to the production of substantial amounts of news articles provided in their regional languages including Telugu, Tamil, Hindi, Bengali, Kannada, and Malayalam. Unfortunately, current systems for processing this data do so independently, rather than as part of a comprehensive framework utilizing all the necessary components in one queue of processing pipelines; OCR extraction, Translation, Summarization, and Bias Detection must all be completed one at a time, and do not allow for seamless data passing between functions. In this paper, we will present Bharat Sum, a multilingual news summarization and bias detection system that incorporates OCR capabilities into five different types of processing stages; OCR Text Extraction using Tesseract, Automatic Language Detection using LangDetect, Topic Segmented Abstractive Summarisation using mT5, Translation to English using mBART, and Sentiment based Bias Classification using DistilBERT - all accessible through a single scalable architecture running on commodity hardware and implemented via Streamlit. Our testing involved 150 news articles covering each of the five languages named above. The results achieved were as follows; OCR extraction accuracy of 89.7%, Language Detection Accuracy of 94.2%, Summarisation Quality (ROUGE-L F1) of 92.1% Translation consistency of 91.4% and Sentiment Classification Accuracy of 88.6%. The average end-to-end processing time was between 10 and 16 seconds. Our analysis of Bharat Sum has revealed that it significantly outperforms previous single function systems by providing an Integrated, Real-Time Multilingual Processing capability which currently does not exist in this context. Bharat Sum has the potential to address significant gaps in the research literature regarding the analysis of Integrated Multilingual Media, and will likely serve as an economically viable solution for organisations conducting Digital Journalism, Media Monitoring, or Accessing Multilingual Information.
Farooq Sunar Mohammad, E.Sneha, B.Kavya et al.· 2026 7th International Confe...· 0 citations
MameLoshnLM, the first open-source 8B-parameter language model built specifically for Yiddish, is presented, providing both a foundation for Yiddish NLP and a practical template for language model development in historically rich but digitally underrepresented languages.
Uri Katz, Omer Goldman, Tomasz Limisiewicz et al.· 0 citations
Stacking multiple imbalance-mitigation techniques on top of a pretrained transformer is widely assumed to compound their individual benefits, yet rigorous component-wise evidence for this assumption remains scarce in the Indonesian text classification literature. Four classification architectures are compared in this work on a publicly available Indonesian news title corpus. The working set contains 27,266 short headlines, drawn as a 30% stratified subsample from a cleaned corpus of 90,891 headlines, spread over nine target categories with a class ratio of 13.29. Three reference architectures are constructed: an LSTM trained from scratch with Random Oversampling, a bidirectional LSTM augmented with additive attention, and a fine-tuned IndoBERT on the oversampled training partition. A fourth architecture extends IndoBERT through three additions, namely learned attention pooling over contextual token embeddings, focal modulation applied on top of the cross-entropy term, and minority-class paraphrasing via Indonesian–English–Indonesian back-translation. Every configuration is evaluated through stratified 5-fold cross-validation, paired t-tests with Bonferroni correction across three comparisons, and McNemar tests on the held-out partition. The fine-tuned IndoBERT with Random Oversampling alone reaches the highest macro F1 of 0.837. By contrast, the combined configuration drops to 0.799, and statistical verification confirms that the gap is systematic rather than attributable to fold-level variation. A component-wise ablation isolates focal modulation as the principal driver of the decline, because it disturbs an already-balanced training distribution. The principal outcome of this study is empirical evidence indicating that composing several imbalance-oriented techniques on a pretrained transformer can yield adverse interactions rather than cumulative gains.
LiteLLM is introduced, a lightweight transformer architecture explicitly optimized for short-text scenarios that delivers competitive performance, fast convergence, and competitive cross-domain performance across heterogeneous short-text settings.
Hussein Ala’a Alkaabi, Fuqdan A. Al-Ibraheemi, Ali kadhim Jasim· Neural computing & applicati...· 0 citations
Overall, this work demonstrates that incorporating explicit linguistic information, including language identity, sentiment polarity, and intensifier information, improves sentiment classification of Gujarati–English code-mixed text.
Chirag D. Shah, Shailesh A. Chaudhari· International journal of com...· 0 citations