A SIMULATION STUDY OF INDONESIAN SENTIMENT CLASSIFICATION FOR A WHATSAPP STUDENT SUPPORT CHATBOT

Abstract

Short Indonesian messages may help institutions organise student support requests, but sentiment polarity does not directly measure psychological well-being. This study evaluates a text-classification component intended for a WhatsApp student support chatbot through a fully synthetic, reproducible simulation. A constructed corpus contains 360 messages from 60 scenario families across six support domains. Families, including their positive and negative variants, are separated into training, validation, and test partitions of 216, 72, and 72 messages. A further 48 synthetic messages test negation, abbreviations, language mixing, and explicitly mixed sentiment. Term frequency–inverse document frequency features and multinomial Naïve Bayes are compared with a constant baseline and logistic regression. On the held-out synthetic test set, Naïve Bayes achieves accuracy of 0.875 and macro-F1 of 0.875, matching logistic regression on both aggregate metrics. On the challenge set, its accuracy decreases to 0.813 and macro-F1 to 0.806. Negation is the weakest scenario, with six correct predictions out of twelve; all six positive negation examples are misclassified. Feature inspection links selected errors to polarity-bearing words and unseen abbreviations. The results establish a reproducible computational baseline and expose limitations obscured by aggregate scores. They do not validate real student conversations, live WhatsApp integration, stress detection, or improvements in well-being.