Sentiment Analysis of JAKI App Reviews Using the Data Imbalance Technique
TL;DR
Overall, the findings demonstrate that different data imbalance handling techniques and hyperparameter configurations influence the performance of the fine-tuned IndoBERT model, with Class Weighting providing the best overall performance, followed by SMOTE and Near Miss.
Abstract
The JAKI (Jakarta Kini) application is a digital public service platform developed by the DKI Jakarta Provincial Government that has received thousands of user reviews on the Google Play Store. These reviews contain diverse opinions, making aspect-based sentiment analysis (ABSA) essential for capturing users’ perceptions of a specific aspect in greater depth. This study develops a predefined aspect-based sentiment classification model for JAKI app reviews by classifying them into three predefined aspects, functional, service, and usability, with three label sentiments, positive, negative, and neutral. The annotation process involved 3,357 reviews and was conducted by five annotators, consisting of two human annotators and three AI-based annotators, resulting in an average Krippendorff’s Alpha of 0.75. The modeling employed XGBoost and a fine-tuned IndoBERT, together with hyperparameter optimization and three data imbalance handling techniques: Near Miss, SMOTE, and Class Weighting. The evaluation results show that the fine-tuned IndoBERT model with Class Weighting achieved the highest overall average macro F1-score. The service and usability aspects achieved the highest macro F1-score of 76%, while the functional aspect recorded the lowest score, 49%, primarily due to severe class imbalance. Overall, the findings demonstrate that different data imbalance handling techniques and hyperparameter configurations influence the performance of the fine-tuned IndoBERT model, with Class Weighting providing the best overall performance, followed by SMOTE and Near Miss.