Beyond Model Complexity: A Reproducible Comparison of Classical Machine Learning, Matrix Factorization, Graph Embeddings, and LightGCN for Recommendation

Abstract

Recommender systems increasingly incorporate graph embeddings and graph neural networks to capture high-order relationships between users and items. However, the additional complexity of these approaches does not necessarily guarantee better recommendation quality than strong classical and latent-factor baselines. This study presents a reproducible comparison of six recommendation models representing four methodological families: Logistic Regression and Random Forest; Matrix Factorization with Bayesian Personalized Ranking; DeepWalk and node2vec; and LightGCN. The experiments were conducted on the MovieLens 1M dataset using a per-user temporal split. For each user, the most recent positive interaction was assigned to testing, the preceding interaction to validation, and all earlier positive interactions to training. The primary evaluation used identical candidate sets containing one held-out positive movie and 99 sampled unobserved movies. Performance was measured using Recall, Precision, Hit Rate, and NDCG at multiple cutoffs, complemented by bootstrap confidence intervals, paired statistical tests, computational-efficiency measurements, and analyses by user activity and movie popularity. Matrix Factorization achieved the best overall performance, reaching a Recall@10 of 0.7458 and an NDCG@10 of 0.4558, representing an approximately 56% improvement in NDCG@10 over Random Forest, the strongest classical baseline. Validation-based tuning improved LightGCN to an NDCG@10 of 0.2875; it significantly outperformed Logistic Regression but remained statistically indistinguishable from Random Forest after Holm correction. Tuned node2vec also significantly outperformed DeepWalk, reaching an NDCG@10 of 0.1593, although both random-walk embedding methods’ results remained substantially below than the strongest baselines. Popularity-based analysis further revealed that classical models and LightGCN achieved substantially higher ranking effectiveness for popular movies, whereas Matrix Factorization maintained comparatively stronger performance for less-popular items. These findings show that under the evaluated setting, greater model complexity did not consistently translate into higher recommendation effectiveness, and they thus highlight the importance of strong baselines, model tuning, standardized evaluation, and reproducible experimental protocols.