TL;DR
LiteLLM is introduced, a lightweight transformer architecture explicitly optimized for short-text scenarios that delivers competitive performance, fast convergence, and competitive cross-domain performance across heterogeneous short-text settings.
With the widespread growth of digital platforms, online interaction has become an essential part of everyday life. Users frequently express their opinions, feedback, and emotions through reviews and comments on various platforms. Analyzing such textual data plays a crucial role in understanding user sentiment and supporting effective decision-making. However, sentiment analysis faces several challenges, including long-range dependencies within text and the presence of unknown words and symbols. Traditional sentiment analysis approaches mainly rely on sequential models, which process text step by step and often require higher computational time. In contrast, Transformer-based models offer improved efficiency through parallel processing. To address these challenges, this paper presents a context-aware hybrid deep learning approach by integrating the Robustly Optimized BERT Pretraining Approach (RoBERTa) with Bidirectional Long Short-Term Memory (BiLSTM) networks. RoBERTa is employed to generate rich contextual word embeddings, while BiLSTM captures long-term semantic dependencies by processing text in both forward and backward directions. The proposed model is trained and evaluated on the Twitter US Airline Sentiment dataset comprising 14,299 samples across three sentiment classes. Experimental analysis demonstrates that the hybrid approach achieves an accuracy of 85.14% and an F1-score of 0.8487, highlighting its effectiveness for sentiment analysis tasks compared to baseline models
Short, informal text on Twitter has become a major source of public-opinion data, yet extracting dependable sentiment signals from it is challenging because many existing models fail to jointly capture fine-grained word-level cues and broader contextual meaning. This study proposes a hybrid deep-learning framework that combines convolutional feature extraction with a bidirectional recurrent encoder and an attention mechanism to categories tweets as positive, neutral, or negative. Three parallel one-dimensional convolutional (Conv1D) branches first capture local n-gram patterns of varying width, after which a Bidirectional Long Short-Term Memory (BiLSTM) layer models dependencies across the whole sequence in both directions, and an additive attention component then highlights the specific tokens most responsible for the predicted polarity. The processing pipeline further incorporates a Twitter-specific preparation stage that performs emoji-to-text conversion, informal-language normalization, and hash tag decomposition, addressing noise patterns that generic NLP pipelines are not designed to handle. The architecture was evaluated on a curate collection of 50,000 manually annotated tweets spanning several topical domains. It achieved a test accuracy of 87.7% and a weighted F1-score of 0.877, outperforming a TF-IDF/SVM baseline (79.8%), a standalone CNN (85.1%), a single-direction LSTM (83.5%), a BiLSTM without attention (85.5%), and a fine-tuned BERT-base model (86.9%). The proposed model maintained consistent performance across politics-, e-commerce-, and health-related tweet subsets, indicating that the learned representations generalize reasonably well across domains rather than over fitting to a single topic. These findings suggest that combining local feature extraction, bidirectional context modeling, and attention-based token weighting can approach transformer-level accuracy while remaining considerably lighter computationally, making the approach attractive for real-time or resource-constrained deployment. Future work is outlined around integrating transformer-based embeddings, extending the framework to multilingual and code-switched text, and adapting the pipeline for continuous streaming inference.
Stacking multiple imbalance-mitigation techniques on top of a pretrained transformer is widely assumed to compound their individual benefits, yet rigorous component-wise evidence for this assumption remains scarce in the Indonesian text classification literature. Four classification architectures are compared in this work on a publicly available Indonesian news title corpus. The working set contains 27,266 short headlines, drawn as a 30% stratified subsample from a cleaned corpus of 90,891 headlines, spread over nine target categories with a class ratio of 13.29. Three reference architectures are constructed: an LSTM trained from scratch with Random Oversampling, a bidirectional LSTM augmented with additive attention, and a fine-tuned IndoBERT on the oversampled training partition. A fourth architecture extends IndoBERT through three additions, namely learned attention pooling over contextual token embeddings, focal modulation applied on top of the cross-entropy term, and minority-class paraphrasing via Indonesian–English–Indonesian back-translation. Every configuration is evaluated through stratified 5-fold cross-validation, paired t-tests with Bonferroni correction across three comparisons, and McNemar tests on the held-out partition. The fine-tuned IndoBERT with Random Oversampling alone reaches the highest macro F1 of 0.837. By contrast, the combined configuration drops to 0.799, and statistical verification confirms that the gap is systematic rather than attributable to fold-level variation. A component-wise ablation isolates focal modulation as the principal driver of the decline, because it disturbs an already-balanced training distribution. The principal outcome of this study is empirical evidence indicating that composing several imbalance-oriented techniques on a pretrained transformer can yield adverse interactions rather than cumulative gains.
The work provides a reproducible, explainable, operationally applicable model of sentiment analysis in operationally sensitive, high-stakes Twitter sentiment analysis, and validate the hypothesis that hybrid stacking is an effective method for leveraging the complementary nature of lexical and contextual representations.
This work evaluates a fixed-list routing strategy that keeps stronger languages on a direct multilingual path and selectively sends weaker languages through translation into English before zero-shot classification, and reports routing through tier-level quality gains and tier-level latency rather than a single global efficiency score.
A hybrid detection framework which combines semantically deep embeddings from the RoBERTa transformer with a set of carefully designed language statistics and linguistic statistics and shows excellent resistance to the surface-level adversarial paraphrasing strategy.