Sep 2026· Proceedings of the Thirty-Fifth International Joint Conference on Artificial Intelligence· pp. 8278-8283· 5 citations· 38 references
TL;DR
This work proposes extending top-1 evaluation to list-wise top-k protocols that assess explanation effectiveness across multiple highly ranked outputs, and shows that top-k evaluation substantially improves consistency and yields more reliable comparisons between competing explainers.
Abstract
Counterfactual explanations have become an important paradigm for improving the transparency of machine learning models by showing how small input changes can alter model outputs. While substantial progress has been made in generating such explanations, their evaluation remains insufficiently standardized, particularly for systems that produce ranked outputs rather than single-label predictions. Existing evaluation protocols in recommender systems commonly focus only on whether the top-1 recommendation changes after perturbation. We argue that this practice can lead to inconsistent and misleading conclusions, as the relative ranking of explanation methods may vary with changes in the quality of the underlying model.
In this work, we revisit the evaluation of counterfactual explanations from a ranking perspective. We propose extending top-1 evaluation to list-wise top-k protocols that assess explanation effectiveness across multiple highly ranked outputs. Through experiments on multiple datasets, recommender architectures, and explanation methods, we show that top-k evaluation substantially improves consistency and yields more reliable comparisons between competing explainers. Our findings highlight a broader methodological lesson for explainable AI: when models return ranked results, explanation quality should be assessed using ranking-aware evaluation protocols rather than top-1 criteria alone.
Recommender systems are increasingly required to satisfy fairness and transparency constraints. However, the interaction between calibration-based fairness interventions and recommendation explainability remains insufficiently understood. This paper proposes a calibration-driven framework that extends single-category c...
P. Atauchi, A. L. Zanon, Leonardo Rocha et al.· Brazilian Symposium on Datab...· 0 citations
Large Language Models (LLMs) are increasingly used to evaluate recommendation systems, but are known to exhibit systematic biases. We study whether LLM judges are influenced by descriptions of a recommendation algorithm’s optimization objective, even when evaluating identical recommendation outputs. We find that they a...
Himan Abdollahpouri, Kyle Kretschman, M. Lalmas· Proceedings of the 20th ACM...· 0 citations
This work presents a general evaluation framework that enhances observability across multiple recommender systems at Netflix and demonstrates its effectiveness through several production deployments, and articulate three stakeholder-centered observability principles for content creators and model developers.
Chao-Ran Guo, Ding Tong, Ting-Po Lee et al.· arXiv.org· 0 citations
TRACE (Targeted Ranking-Aware Counterfactual Explanation), which decomposes the search into three stages under an embedding-accessible, training-free setting: influence-guided position selection, plausibility-aware candidate retrieval, and actual-margin beam pruning.
Ungsik Kim, Sang-Min Choi, Gun-Woo Kim et al.· Proceedings of the 20th ACM...· 1 citation
This reproducibility paper explores the counterfactual explanation technique known as Learning to eXplain Recommendations (LXR), and considers alternate baselines for LXR, including item popularity and latent factor similarity, and finds that for some tasks these alternatives work surprisingly well.
Meysam Varasteh, Elizabeth McKinnie, Robin D. Burke· 0 citations
Comprehensive evaluations of MAREval establish MAREval as a scalable and reliable solution for human-aligned evaluation of recommendation explanations in real-world systems and improves alignment with human judgments over strong singleand multi-agent baselines.