The XAI analysis reveals that interpreting the VAE-based model’s decision uncovers stylistic differences between the two text types beyond superficial lexical and syntactic features, demonstrating the potential of the XAI DL framework in other NLP studies that aim to analyze style.
Abstract
Text classification using traditional machine learning techniques has been used in natural language processing (NLP) tasks to distinguish translated from non-translated languages, with high accuracy scores indicating the distinctive style of translated languages. While deep learning (DL) has demonstrated impressive performance in terms of representation learning and capturing nuanced patterns in natural language data, DL models act as black boxes, making their results difficult to interpret. This study addresses this issue by demonstrating an explainable AI (XAI) DL framework in a case study of United Nations (UN) meetings. The framework consists of three stages: i) train a variational autoencoder (VAE) combined with BERT embeddings converted from translated and non-translated texts; ii) utilize the majority vote from three classifiers selected from a stacked ensemble to classify the VAE’s latent representations; iii) implement a perturbation-based XAI method to interpret the DL model’s decisions. The results indicate that the VAE-based model effectively distinguishes the two text types, with accuracy scores above 0.8. The XAI analysis reveals that interpreting the VAE-based model’s decision uncovers stylistic differences between the two text types beyond superficial lexical and syntactic features. This proof-of-concept study demonstrates the potential of the XAI DL framework in other NLP studies that aim to analyze style.
A hybrid detection framework which combines semantically deep embeddings from the RoBERTa transformer with a set of carefully designed language statistics and linguistic statistics and shows excellent resistance to the surface-level adversarial paraphrasing strategy.
Anita Rani, Suman· International Journal of Sci...· 0 citations
Pretrained language models (PLMs) have established state-of-the-art performance across diverse natural language understanding (NLU) tasks. This study reveals that seman-tic-rich explanations of lexical units can effectively guide PLM learning processes. We propose a novel language understanding enhancement method with token interpretation (LUETI) that addresses two critical limitations in conventional PLMs: Incomplete token semantics caused by isolated contextual learning and insufficient semantic encoding in embedding matrices. LUETI operates through dual mechanisms, augmenting token represen-tations by integrating hidden states with corresponding token interpretations and refining embedding spaces using interpretation-derived semantic vectors for token prediction. LUETI, which is implemented as a plug-in module for standard architectures, demonstrates significant improvements on BERT and GLM, achieving average performance gains of 3.36% and 4.87% respectively on the SuperGLUE benchmark with equivalent parameters and training data. Note that LUETI-equipped models attain comparable performance to baseline PLMs using only 60% of pretraining data. Findings establish token interpretation as a computationally efficient but semantically powerful enhancement strategy for language model pretraining.
Tianyi Chen, Yashen Wang, Huan Chang et al.· IEEE/CAA Journal of Automati...· 0 citations
This study evaluates the systematic evolution and computational adaptation of pre-trained language models and Large Language Models (LLMs) for Bahasa Indonesia and its low-resource regional dialects. Initially centered on bidirectional encoder-based representations like IndoBERT, the regional natural language processing (NLP) field has transitioned toward generative sequence-to-sequence structures and massive decoder-only architectures. This paper investigates the engineering methodologies of cross-lingual vocabulary adaptation, parameter initialization heuristics, and language-adaptive pre-training strategies designed to address text overfragmentation, representational misalignment, and tokenization cost inefficiencies. Through extensive structural benchmarks, this analysis compares discriminative and generative performances across tasks including sentiment classification, extractive question answering, text style normalization, domain-specific retrieval-augmented pipelines, and entity linking. While localized generative models such as Komodo, Sailor, and the SEA-LION suite improve contextual reasoning, colloquial style transfers, and regional dialect preservation, they remain susceptible to architectural anomalies like template leakage and entity hallucination. This study provides foundational benchmarks and methodological frameworks for adapting massive language models to morphologically rich, culturally diverse, and low-resource linguistic environments.
A. D. Alexander, S. Setiawati· Dinasti Information and Tech...· 0 citations
India's linguistic landscape, comprising more than twenty scheduled languages and hundreds of additional dialects spanning multiple language families, presents a distinctive and severe challenge for machine translation (MT) systems predominantly developed and benchmarked on high-resource, Indo-European languages. This paper reviews the evolution of AI-driven natural language processing (NLP) approaches to Indian vernacular languages, tracing the shift from rule-based and statistical syntactic methods toward transformer-based semantic representation learning. The review synthesizes the transformer and multilingual pretraining literature, corpus-development efforts specific to Indian languages, and the growing evidence base on cross-lingual transfer and low-resource neural machine translation (NMT). Particular attention is given to the structural and morphological divergence between Indian languages and the English-centric architectures on which most large language models are trained, and to recent large-scale parallel-corpus and translation-model initiatives targeting this gap directly. Comparative tables summarize corpus scale, language coverage, and reported translation-quality metrics across the reviewed systems. The paper concludes that dedicated multilingual pretraining and large-scale parallel-corpus construction, rather than generic multilingual scaling alone, are the primary drivers of translation-quality gains for Indian vernacular languages, and identifies dialectal and code-mixed language coverage as the central future research prospect.
Dr. R. Sugunthakunthalambigai, Dr. Mallanna Biradar, Dr. Joyir Siram et al.· Anusandhanvallari· 0 citations
Plain Language seeks to enhance the clarity and comprehensibility of legal and administrative communication; while Natural Language Processing (NLP) offers promising tools for assessing text complexity, most Plain Language classification studies focus exclusively on English, leaving low-resource languages underexplored. This study presents the first systematic evaluation of transformer-based models for sentence-level Plain Language classification in Hungarian tax administrative texts. We benchmarked zero-shot prompting with GPT-4o against fine-tuned open-weight and proprietary models, including huBERT, XLM-RoBERTa, GPT-4o-mini, and Gemini 1.0 Pro, and contextualized these results against previously established lightweight machine learning baselines based on term frequency-inverse document frequency with a support vector machine (TF-IDF + SVM) and fastText. To address data scarcity, we applied translation-based data augmentation using parallel Hungarian–English corpora. The best-performing model achieved a macro-average F1-score of 0.79. Mid-sized models also delivered competitive results, combining accuracy with feasible inference speed and deployment flexibility. Beyond classification performance, we conducted local and aggregated interpretability analysis based on Shapley-values to identify linguistic patterns influencing model decisions. This revealed alignment with known Plain Language features, such as nominalizations and syntactic complexity, as well as biases introduced by frequent domain-specific terms. Our findings demonstrate that Plain Language classifiers can be effectively adapted to low-resource legal–administrative domains. The results support the development of real-time feedback tools that promote linguistic accessibility and contribute to the broader goal of Access to Justice.
Transformer-based language models have become the standard in Natural Language Processing (NLP). They have surpassed human performance on specific classification tasks such as named-entity recognition, question-answer, text categorization, or generative tasks such as machine translation and summarization. However, since language models are trained with significant general-purpose texts, they may have limitations in their domain-specific knowledge. Techniques such as domain adaptation can be used to improve the models to address this issue. This study presents a systematic empirical investigation of task-adaptive continual pre-training (TAPT), introduced by Gururangan et al., for Turkish language understanding, with a particular focus on the effect of the masked-language-modeling rate. Adaptation is performed in the task-adaptive setting (TAPT), i.e., continual pre-training on the unlabeled text of the target task corpus, without requiring an external domain corpus. We achieved successful results with an average increase of 2.7%. We also addressed various issues and findings related to adaptation.
Murat Aydoğan, Savas Yildirim, Tuǧba Dalyan· IEEE Access· 0 citations