Explanation Consistency as a Trust Signal in Human–AI Teaming

Abstract

Post-hoc explanation methods such as SHAP and LIME are widely used in human-AI decision support, yet they can produce inconsistent explanations for the same prediction under minor perturbations, changes in background samples, or different random seeds. This instability raises concerns about whether explanations provide a reliable basis for user trust and reliance. We introduce a framework for quantifying and reducing explanation inconsistency in tabular decision-support settings. First, we propose a distributional consistency metric based on Jensen–Shannon distance that measures agreement across repeated explanation runs and explanation methods. Second, we introduce Explanation Stability Regularization (ESR), a consistency-aware regularization and model-selection approach that penalizes attribution variability while monitoring predictive performance. Finally, we evaluate the relationship between explanation consistency and human-AI teaming outcomes through a controlled study using tabular classification tasks from the UCI Adult Income and COMPAS datasets. Across three model classes, ESR improves explanation consistency while maintaining comparable predictive accuracy in the tabular settings studied here. Participants exposed to more consistent explanations report higher trust, make more accurate decisions, and show better-calibrated reliance in the controlled study. These findings suggest that explanation consistency can serve as a useful trust signal for tabular human-AI decision support, while broader generalization to expert users, high-stakes settings, and higher-dimensional domains remains an important direction for future work.