On the Stability of SHAP Explanations with Incremental User Data: A Case Study in AI-Powered Fitness Apps

Abstract

Abstract—Abstract—Explainable AI methods such as SHAP,which is a tool for generating Shapley Additive explanations are widely used in health and fitness applications to give reasons for predictions , but their reliability is usually studied on static datasets. Most fitness apps collect user data incrementally, leading to a cold-start problem: can we trust the explanations when only a few sessions are available? We simulated this scenario by training an XGBoost model on synthetic workout logs from 10 users, with 50 sessions for each. We collected SHAP explanations at 5-session intervals and evaluated the stability of feature rankings using Jaccard similarity and Kendall’s tau. Results showed that the top-5 most important features were surprisingly stable from the first sessions (mean Jaccard similarity 0.80 ± 0.17 from session 5 to 10 and remained consistent through session 50. However, the complete feature ranking, measured by Kendall’s tau, was close to zero (±0.33) in the beginning and only reached 0.07 (±0.46) by session 45 to 50, indicating that lower-importance features continued to reorder as more data accumulated. These results suggest that while the factors with the most influence are identified quickly, the complete explanation is still sharpening after 50 sessions. Therefore, fitness apps could safely highlight the key drivers of a prediction even for newer users, but should consider that the detailed feature ranking may evolve over time. It is recommender that fitness apps highlight key factors immediately while cautioning that the full explanation is still evolving.