Designing for Predictability: How Mental Models of AI Error Boundaries Shape Human-AI Team Performance in High-Stakes Decision Making

Abstract

As artificial intelligence systems are increasingly deployed to advise human decision-makers in high-stakes domains—including healthcare, criminal justice, and financial services—organizations face a critical but often overlooked challenge: the accuracy of an AI system alone does not determine the performance of the human-AI team. This article examines the role of human mental models of AI capabilities, specifically mental models of AI error boundaries, in shaping collaborative decision-making outcomes. Drawing on foundational experimental research demonstrating that properties such as the parsimony and stochasticity of an AI's error boundary significantly influence whether humans can learn when to trust or override AI recommendations, this article translates those findings into actionable organizational strategies. Evidence-based interventions are presented across interface design, model selection, training protocols, and governance structures, illustrated with examples from healthcare, surgery, criminal justice, and AI research and development. Forward-looking pillars for building long-term human-AI collaboration capability—including psychological contract recalibration, continuous learning systems, and distributed oversight—are proposed for practitioners seeking to maximize the return on AI-augmented decision systems.