Reliability Evaluation of Large Language Models for Social Media Sentiment Annotation: An Empirical Study Based on Model Agreement and Downstream Tasks
Jul 2026· Computers and artificial intelligence· 0 citations· 14 references
TL;DR
Obvious performance discrepancies exist across LLMs; high-quality annotations significantly boost downstream classification accuracy, which highlights the importance of selecting competent LLMs for sentiment labeling tasks.
Abstract
Driven by booming social media and user-generated texts, sentiment analysis stands as a core natural language processing task, yet large-scale high-quality data labeling comes with steep costs and practical barriers. This work assesses how dependable large language models are for sentiment tagging, alongside how their labeled outputs shape subsequent classification effects. We build a dataset containing 1,543 Chinese entertainment comment snippets scraped from Bilibili. Under unified prompting, three LLMs—DeepSeek, Qwen and Doubao—produce zero-shot sentiment tags, while 500 sampled entries receive manual annotation to form authoritative benchmark labels. Cohen’s Kappa is adopted to quantify human-model annotation consistency, and TF-IDF features are input to Logistic Regression, Linear SVM and Random Forest for downstream classification evaluation. Among the three models, Doubao achieves the highest human-label consistency with a κ value of 0.800, exceeding Qwen (κ=0.746) and DeepSeek (κ=0.636). Classifiers trained on Doubao’s labels obtain optimal Macro-F1 values of 0.8125 (SVM) and 0.8174 (Random Forest). Obvious performance discrepancies exist across LLMs; high-quality annotations significantly boost downstream classification accuracy, which highlights the importance of selecting competent LLMs for sentiment labeling tasks.
. Social media sentiment analysis faces a persistent aggregation problem: lexicon-based and transformer-based models often produce inconsistent outputs for the same short, informal, and stylistically heterogeneous texts. This paper introduces ADRTW (Adaptive Dynamic Reliability-Trig-gered Weighting), an interpretable sentiment fusion framework that combines heterogeneous sentiment estimators using rule-guided reliability weights derived from textual cues, inter-model disagreement, and consistency patterns [5, 8]. The framework is evaluated on a Reddit dataset containing 1,577 posts, 354,050 comments, and 187,666 authors collected between 2017 and 2025, together with a controlled synthetic benchmark for aggregation comparison. The results show that ADRTW remains competitive with static averaging in controlled settings while preserving context-sensitive local variation in large-scale discourse analysis. Beyond sentiment fusion, the ADRTW-derived signal supports complementary analyses of online discussions, including temporal trend inspection, toxicity-aware interpretation, and participation-based clustering. Overall, the proposed framework provides a transparent and reusable basis for examining emotional dynamics in social media discourse.
Aniko Apro, L. Sasi· Annales Mathematicae et Info...· 0 citations
Sentiment analysis (or opinion) is a core problem in Natural Language Processing (NLP) that aims to automatically determine the polarity of opinions expressed in text. As online review platforms continue to expand, the volume of user generated content has grown beyond what can be examined manually. Movie reviews are particularly challenging because they are subjective, vary widely in length, and use diverse writing styles. This paper presents a sentiment classification pipeline for IMDb movie reviews that combine rigorous text cleaning with Term Frequency Inverse Document Frequency (TF–IDF) feature extraction and classical supervised learning models. Exploratory analysis is used to examine sentiment balance and review-length characteristics. The pre-processing stage removes HTML/URLs, normalizes text, filters stop words, and applies lemmatization to reduce noise and vocabulary sparsity. TF–IDF vectors with unigram and bigram features are then constructed to capture both individual sentiment cues and short contextual phrases. Logistic Regression, Linear SVM, Multinomial Naïve Bayes, and Random Forest are evaluated under a consistent experimental setup. Performance is reported using accuracy, precision, recall, F1-score, and ROC-based analysis. The results show that Logistic Regression achieves the best overall performance, with an accuracy of 88.68% and an F1-score of 88.62%, followed closely by Linear SVM. Overall, TF–IDF with linear classifiers remains an interpretable and computationally efficient baseline for practical sentiment classification.
Pooja Limbola, Abhijit Dwivedi, Namrata Shrivastava et al.· 2026 International Conferenc...· 0 citations
The rapid growth of online shopping platforms has significantly increased the analytical value of product review data. Short reviews on Chinese e-commerce platforms, typically containing fewer than 50 characters, simultaneously address multiple dimensions, including product quality, logistics, customer service, and value for money, rendering conventional sentence-level sentiment classification insufficient for fine-grained analysis. This study adopts aspect-based sentiment analysis (ABSA) as the core framework and constructs a labeling schema comprising four sentiment dimensions and three polarity levels, yielding 12 classification categories. Rule-based automated annotation was applied to 62,770 publicly available Chinese consumer reviews; manual verification on a 500-sample subset confirmed an annotation accuracy of 87.2%. Three lightweight models—TextCNN, BiLSTM with attention (BiLSTM+Attn), and ERNIE 3.0-Nano—were trained and evaluated under identical experimental conditions on an NVIDIA RTX 4060 laptop GPU. ERNIE 3.0-Nano proved to be the most accurate, with an accuracy of 0.6363 and a macro-average F1 score of 0.5798. BiLSTM+Attn was the fastest on GPU: 0.004 ms per sample. TextCNN had the smallest parameter count: 0.69 million. These findings provide quantitative evidence for model selection under diverse deployment constraints.
Jiaye Huang· Applied and Computational En...· 0 citations
This paper presents a framework that preserves semantics in LLM-based opinion summarization while minimizing token usage and computational cost and demonstrates that this method significantly reduces token usage and computational cost while consistently outperforming traditional AI-based and standard LLM summarization baselines in terms of content coverage, balance, and semantic preservation.
The widespread adoption of social media platforms has increased the availability of code-mixed textual data, particularly for low-resource languages such as Marathi–English. But sentiment analysis has considerable obstacles stemming from multilingual diversity, transliteration complexity, and the scarcity of high-quality annotated datasets. This work presents `KonkaniFood 2.0`, a 5,195 YouTube comments dataset of Konkani cuisine, collected through web scraping and carefully annotated for positive, negative and neutral sentiment (3-Class), with a Fleiss's kappa score of 0.967. Sentiment categorisation was carried out using Advanced Transformer-Based Language Models: mBERT, MuRIL,and IndicBERT. MuRIL model showed a better understanding of the semantic nuances of code-mixed text with 98% accuracy. Moreover, explainable AI (XAI) approaches were employed to interpret the model predictions by highlighting sentiment-carrying phrases, hence boosting transparency and trustworthiness. This work offers a valuable resource in Marathi-English (Mr-En) resource-scarce, code-mixed multilingual sentiment analysis.
A. Phadke, S. Kohle, Praveen Gupta· 2026 4th International Confe...· 0 citations
The work provides a reproducible, explainable, operationally applicable model of sentiment analysis in operationally sensitive, high-stakes Twitter sentiment analysis, and validate the hypothesis that hybrid stacking is an effective method for leveraging the complementary nature of lexical and contextual representations.
D. Abate, Nilay Mistry· International Research Journ...· 0 citations