It is argued that cognitive misalignment represents a likely impediment to AI adoption in many envisioned applications, and that addressing it is important for creating AI systems on which users are both willing and justified to rely.
Abstract
AI systems are increasingly employed as decision aids, decision delegates, or autonomous decision-makers. This position paper argues that in many settings, particularly high-stakes decision-making, we need accurate cognitively-aligned AI systems that reason similarly to their users, and faithfully communicate their reasoning. We review evidence that cognitive alignment improves understandability and trustworthiness, and provide new survey data showing that many users find cognitive alignment"essential"when an AI's rationale for a judgment or action is important to them. We outline the gaps between existing alignment methods and what is needed to achieve cognitive alignment, and present a research agenda to address these gaps. We argue that cognitive misalignment represents a likely impediment to AI adoption in many envisioned applications, and that addressing it is important for creating AI systems on which users are both willing and justified to rely.
It is argued that intelligence alone cannot determine appropriate goals, guide action under uncertainty, or ensure beneficial human outcomes, and proposed artificial wisdom as a needed corrective for conversational systems.
Matthias Kraus· International Conference on...· 0 citations
It is argued that psychological competence should become a core consideration for model providers, deploying organizations, researchers, and regulators concerned with the real-world effects of human-facing AI systems.
M. Economides, Paul M. Sacher, Samuel Salzer et al.· 0 citations
This article addresses the growing concern that generative artificial intelligence (AI) may replace human expertise in organizations. Instead of asking whether AI should be used, it examines why human judgment rooted in experience cannot be fully replaced by current AI systems and how organizations can work with AI more effectively. Drawing on research from cognitive science, neuroscience, and organizational studies, the paper explains how people use prior experience to interpret context, notice subtle cues, and make sense of ambiguous situations—capabilities that differ fundamentally from how large language models process data. Evidence from recent studies of AI use in hiring, performance management, healthcare, and knowledge work shows recurring problems, including mistakes in unusual cases, missed context, over-reliance on AI recommendations, and reduced visibility of real skill differences among employees. In response, we propose a five-part Human–AI Collaboration Framework designed to help organizations use AI for efficiency while keeping human judgment active and accountable in key Human Resource Management decisions. The analysis shows that AI performs best in routine, data-rich situations but falls short when decisions require lived experience and contextual understanding. By framing organizations as systems built on accumulated experience, this article offers practical guidance for responsible AI integration and outlines directions for future research on human–AI collaboration.
Daniel Altieri, Zohra Damani, Cynthia L. Nebel· Administrative Sciences· 0 citations
Recent advances in AI make it conceivable to delegate legal decision-making to machines, or to enhance human adjudication through AI assistance. Using classic normative conflicts — the trolley problem and comparable moral dilemmas — as a proof of concept, we examine the alignment between AI legal reasoning and human judgment. In our baseline experiment, we find a pronounced mismatch between decisions made by GPT and those of human subjects. This misalignment raises substantive concerns for AI-powered legal decision-aids. We investigate whether explicit normative guidance can address this misalignment, with mixed results. is susceptible to such intervention, but frequently refuses to decide when faced with a moral dilemma. is outright utilitarian, and essentially ignores the instruction to decide on deontological grounds. faithfully implements this instruction, but is unwilling to balance deontological and utilitarian concerns if instructed to do so. We replicate the experiment with four LLMs from different providers. comes closest to human respondents. is most sensitive to normative instructions. and have a strong utilitarian bias, and do not strongly respond to normative interventions. At least for the time being, explicit normative instructions are not fully able to realign AI advice with the normative convictions of the population, or the legislator deciding on its behalf.
This position paper argues that human-centered explainable AI (HCXAI) should incorporate insights from the psychology of information seeking. Drawing on Sharot and Sunstein's framework of information-seeking motives, we propose that people evaluate whether to engage with explanations based on three types of expected utility: instrumental (will it help me act better?), hedonic (will it make me feel better?), and cognitive (will it improve my understanding?). Each utility is estimated through a lens shaped by well-documented cognitive biases, including illusion of control, automation bias, unrealistic optimism, impact bias, overconfidence, and confirmation bias. These biases can lead to two failure modes: excessive information-seeking that fragments attention without improving decisions, and insufficient information-seeking that leaves critical risks and misunderstandings unexamined. This challenge is particularly acute for agentic AI systems, where explanations must support not just understanding a single output but anticipating cascading actions, assessing risks, and deciding when to intervene. By integrating information-seeking psychology into HCXAI, we advocate for a shift from making explanations available to making them sought: designing systems that account for when and why users actually want to know.
