The Generative Shortfall: a comparative analysis of autoregressive vs. extractive architectures in ABSA

Abstract

In natural language processing (NLP), Aspect-Based Sentiment Analysis (ABSA) remains a challenge, particularly when handling ambiguous, domain-specific “neutral” sentiment classifications in cases of severe class imbalance. While there has been a pivot toward deploying generative Small Language Models (SLMs) on resource-limited edge devices, their performance on unevenly distributed data remains under-explored. This paper presents a comprehensive, cross-paradigm evaluation of the following ABSA architectures: a sequence-based BiGRU, a state-of-the-art DeBERTa-v3 encoder, and Alibaba’s Qwen2.5-1.5B generative SLM. To evaluate the feasibility of deploying the SLM on edge devices, we modified the Qwen2.5 model to work on a 6GB VRAM device using BFloat16 quantized low-rank adaptation (QLoRA) and an abstract syntax tree (AST) parser. A closer look at the results shows a fundamental Generative Shortfall. Despite optimizing the SLM for edge devices, its performance was suboptimal (accuracy: 79.67%, macro-F1: 0.71), indicating that the model’s conservative token generation behavior limited its ability to identify sparse and subtle semantic targets within the text. While the DeBERTa-v3 encoder, when balanced with multi-sample dropout and layer-wise learning rate decay (LLRD), achieved a macro-F1 of 0.87, successfully identifying 81.95% of the minority neutral class. Supported by bootstrap resampling and McNemar’s statistical significance test ( p<0.001 ), the results show that although generative SLMs can be efficiently deployed on resource-constrained edge devices, structurally regularized extractive encoders provide significantly better performance for precise opinion mining on highly imbalanced data.