Evaluating the Performance of Large Language Models in Aspect-Based Sentiment Analysis: Cyprus Hotel Reviews

Abstract

The rapid proliferation of hotel reviews on online travel platforms, such as Booking.com and TripAdvisor, has necessitated the processing of large volumes of textual data for tourism researchers. The majority of existing Sentiment Analysis (SA) studies are limited to single-model approaches, which often lack validation against human-annotated labels and are restricted to document-level analysis. In this study, a structured multi-model framework is proposed for Aspect-Based Sentiment Analysis (ABSA) applied to 56,790 English hotel reviews collected from 30 hotels in Northern and Southern Cyprus. Three distinct generations of AI models are systematically compared: VADER (rule-based), ChatGPT/GPT-4o-mini (Large Language Model, prompt engineering), and DistilBERT (transformer). Eight service quality themes—location, food, service, cleanliness, rooms, facilities, value, and atmosphere—are extracted by generating structured JSON outputs via GPT-4o-mini, and the results are validated against human-annotated labels (achieving 91% agreement on a sub-sample of 100 reviews). The proposed framework demonstrates a scalable and reproducible methodology for aspect-oriented sentiment analysis in the hospitality domain.