Aug 2026· ACM Transactions on Recommender Systems· 0 citations· 37 references
TL;DR
This work systematically analyze how extending evaluation beyond top-1 to top-k recommendations improves metric stability and reduces dependence on recommender fluctuations, and introduces compactness-aware evaluation criteria that quantify the minimality of counterfactual modifications.
Abstract
Explainability in recommender systems (RS) remains a pivotal challenge. Counterfactual explanations have emerged as a particularly actionable paradigm, offering intuitive “what-if” reasoning. However, their evaluation lacks principled standards. Current metrics primarily assess whether explanations change the top-ranked recommendation, overlooking two fundamental aspects of explanation quality. First, evaluation results are frequently inconsistent, as metrics are tightly coupled to the underlying recommender’s performance. Second, explanations are rarely assessed for compactness—whether changes are sufficiently small to remain interpretable. Oversized counterfactuals may technically succeed but fail to provide practical insight. In this work, we advocate for a holistic evaluation perspective centered on consistency and compactness. We systematically analyze how extending evaluation beyond top-1 to top-k recommendations improves metric stability and reduces dependence on recommender fluctuations. In parallel, we introduce compactness-aware evaluation criteria that quantify the minimality of counterfactual modifications. Through extensive experiments across multiple datasets and models, we demonstrate that jointly considering these dimensions yields more reliable assessments. Our findings expose key factors driving metric instability, highlight the trade-off between effectiveness and explanation size, and provide practical guidelines toward standardized, faithful, and compact evaluation of counterfactual explanations in recommender systems.
Recommender systems increasingly influence what users discover and consume online, yet the decision logic behind their outputs often remains opaque. Counterfactual explanation (CFX) methods provide an effective way to show which changes to a user's interaction history would alter a recommendation. In recommendation set...
Vojtech Cahlík, Amir Reza Mohammadi, R. Alves et al.· Proceedings of the 35th ACM...· 0 citations
Recommender systems are increasingly required to satisfy fairness and transparency constraints. However, the interaction between calibration-based fairness interventions and recommendation explainability remains insufficiently understood. This paper proposes a calibration-driven framework that extends single-category c...
P. Atauchi, A. L. Zanon, Leonardo Rocha et al.· Brazilian Symposium on Datab...· 0 citations
Efficiency and trustworthiness constitute two of the pivotal desiderata in modern recommender systems architecture. In this sense, increasingly large and complex models can improve recommendation quality, while explainability mechanisms provide understandable and verifiable justifications for their outputs. However, th...
Leonardo Di Gioia· Proceedings of the 35th ACM...· 0 citations
This reproducibility paper explores the counterfactual explanation technique known as Learning to eXplain Recommendations (LXR), and considers alternate baselines for LXR, including item popularity and latent factor similarity, and finds that for some tasks these alternatives work surprisingly well.
Meysam Varasteh, Elizabeth McKinnie, Robin D. Burke· 0 citations
This work presents a general evaluation framework that enhances observability across multiple recommender systems at Netflix and demonstrates its effectiveness through several production deployments, and articulate three stakeholder-centered observability principles for content creators and model developers.
Chao-Ran Guo, Ding Tong, Ting-Po Lee et al.· arXiv.org· 0 citations
Explanations in Recommender Systems (RS) play a crucial role in enhancing transparency, trust, and user satisfaction. Benchmarking such explanations, however, is not trivial, as existing evaluation frameworks typically do not assess all explanation stages (i.e., generation, abstraction, formatting, and communication) o...
Kathrin Wardatzky, Oana Inel, Abraham Bernstein· Proceedings of the 35th ACM...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.