FASC: A Feature Aspect-Level Sentiment Consistency Framework for Explainable Recommendation Evaluation

Abstract

Recent research in explainable recommendation commonly uses natural language explanations to improve transparency and user trust. However, reliably evaluating whether explanations are semantically faithful to users’ multi-dimensional preferences remains challenging. Existing methods mainly rely on text similarity metrics (e.g., BLEU, ROUGE) or shallow feature matching, which are insufficient for assessing whether explanations accurately reflect user preferences, especially in multi-aspect settings. To address these limitations, we propose Feature Aspect-Level Sentiment Consistency (FASC), a framework that quantifies semantic consistency between generated explanations and user-authored reference explanations through aspect coverage and sentiment polarity. FASC uses LLMs as auxiliary tools to extract structured aspect–sentiment units from explanation texts, enabling reproducible metrics for aspect coverage, correctness, and sentiment alignment. To validate the framework, we re-annotated several widely used explainable recommendation datasets to construct benchmarks with fine-grained, aspect-level sentiment labels. We further conducted human studies using pairwise comparisons, showing that FASC aligns more closely with human judgments than traditional metrics. Experimental results indicate that FASC can distinguish subtle differences in semantic faithfulness across models. To support future research, we release all annotated datasets and human evaluation results at https://github.com/YuChenfu1022/FASC.