Comparative Evaluation of Precision and Relevance in Traditional, Graph, and LLM-Prompted Recommender Models

Abstract

Most of the time, recommendation systems are evaluated using accuracy metrics like RMSE and MAE, but these accuracy values do not guarantee the generation of relevant recommendation lists for users. This study explains the difference by experiment with varied cross-paradigm comparison of User-Based Collaborative Filtering (UBCF), Singular Value Decomposition (SVD), Neural Collaborative Filtering (NCF), LightGCN, and an LLM-Prompted model. Those models are evaluated using RMSE, MAE, Precision@10, and NDCG@10 metrics, and using the same dataset, which is MovieLens 100K. Our findings show that SVD achieves the best RMSE value of 0.9373, but only ranks third in NDCG@10 with a value of 0.2060. Meanwhile, LightGCN records the worst RMSE value of 1.7790, actually the second-highest NDCG@10 of 0.4647, a 125.6% difference compared to SVD. The LLM-Prompted model, operating as a zero-shot ranker without training on interaction data, achieves a Precision@10 of 0.0960 and an NDCG@10 of 0.9397, outperforming all other models. These findings confirm that prediction accuracy and ranking quality are two misaligned objectives. Only evaluation with RMSE values alone for model selection in recommender systems risks overlooking architectures that provide significantly more relevant recommendation lists for end-users. The GitHub link of our experiment and research can be accessed at this link: https://github.com/Hinsane5/Movie_Recommendation.