PhoBERT: Long-Document Sentiment Analysis in Vietnamese with Sparse Attention

Abstract

The rapid growth of user-generated content on Vietnamese e-commerce platforms (Tiki, Google Play) has created an urgent need for accurate sentiment analysis of long documents (>256 tokens). However, existing Vietnamese Transformer models like PhoBERT are limited by the 256-token input limit, leading to a loss of context when emotional signals appear later in the text. This study presents sentiment analysis of long documents in Vietnamese. We introduce a balanced dataset of 30,000 reviews (10,000 of each sentiment type: Positive 33.33%, Neutral 33.33%, Negative 33.33%) stratified by four length groups (<256, 256-512, 512-1024, >1024 tokens). We compared PhoBERT's basic models (truncation, sliding window, hierarchy) with a Longformer model initialized from PhoBERT, expanding the context to 2048 tokens through sparse self-attention mechanisms. Experimental results showed that Longformer achieved a Macro-F1 of 0.9206 (compared to 0.8923 with truncation, +2.83%), with performance increasing positively with document length (F1=0.9975 for >1024 tokens) while the truncation method reduced performance when exceeding 512 tokens. These results confirm that explicit long-context modeling via sparse attention is essential for robust document-level sentiment analysis in Vietnamese and provide the first reproducible framework for adapting monolingual Transformer models to long-document tasks in resource-limited languages.