Cloud-Based Multi-Criteria Evaluation of Conventional and LLM-Based Recommender Systems

Abstract

Recommender systems are a well-established area of the digital ecosystem supporting personalization in a wide range of domains in cloud computing environments. As recommendation functionality is increasingly delivered through cloud infrastructures, factors beyond recommendation accuracy, including latency and operational cost, have become critical for real-world applications. At the same time, Large Language Models (LLMs) have emerged as a promising direction for next-generation recommender systems. The goal of this work is to provide a comparative evaluation of different recommender systems based on trained conventional recommendation algorithms and zero-shot LLMs in a cloud environment with respect to recommendation quality, latency, operational cost and carbon footprint. A total of 18 different recommender systems are compared based on the MovieLens 100K dataset and a 20-item sampled-candidate setting, 13 of which utilize conventional recommendation algorithms while the remaining 5 are zero-shot LLMs. The results confirm the superiority of the neural collaborative model NeuMF and the embeddings-based matrix factorization method BPR-MF. In contrast, LLMs in a zero-shot setting underperformed while reporting a much higher estimated carbon footprint. Overall, this work provides an up-to-date benchmarking of trained conventional and zero-shot LLM-based recommender systems and highlights pivotal tradeoffs that should be considered when deploying recommendation systems in cloud environments.