Skip to content
Preprint

AI advice suppresses people's willingness to say"I don't know", even when the advice is wrong and accuracy is incentivized

Jul 2026 · 2 citations · 43 references
Computer Science

Abstract

Knowing when to say"I don't know"is fundamental to human judgment, yet AI assistants offer a fluent answer to almost any question. In five experiments (N = 3,132; four preregistered, one direct replication), participants answered difficult questions and could always decline to respond. We engineered the questions so that AI advice was wrong, separating AI use from its accuracy. Merely having access to AI nearly eliminated participants'willingness to suspend judgment, and this held whether the advice was actively requested or simply displayed. Consequently, participants answered more questions but were correct about a third as often as when AI was unavailable-yet their confidence nearly doubled. Incentivizing accuracy and penalizing inaccuracy led participants to seek and follow AI advice less, answer more accurately, and suspend judgment more often, though still far less than when AI was unavailable. As AI suggestions grow ubiquitous and unsolicited, they may not simply affect answer accuracy; they may even alter the metacognitive threshold at which people decide whether they know enough to answer.

View source

Similar papers

Preprint Aug 2026

How People Evaluate AI-, Expert-, and Peer-Style Financial Advice

Findings show that financial-advice evaluations are shaped jointly by displayed attribution and message-level communication cues, which position disclosure not as a neutral transparency mechanism, but as an interpretive frame whose accuracy and interaction with message cues can shape trust and reliance.

A. Kapadia, Eshwar Chandrasekharan, Koustuv Saha · 0 citations
Review Jul 2026

Beyond Accuracy: How Humans Evaluate Legally Correct but Socially Controversial Legal Advice from Machines

AI systems are increasingly used to provide legal advice, raising questions about whether laypeople accept guidance from algorithms--especially when that advice is legally correct but socially controversial. We report a preregistered survey experiment with 3,348 adults in mainland China examining how people evaluate identical legal advice when it is attributed either to an AI system or to a human lawyer, and when it is accompanied by reasoning or not. Contrary to expectations of algorithm aversion, attribution to an AI system has no net effect on perceived reasonableness. However, mediation analyses reveal opposing psychological pathways underlying this null result. AI-attributed advice is perceived as more objective, which increases perceived reasonableness, but also as less comprehensive and less attentive to special circumstances, which decreases perceived reasonableness. By contrast, providing legal reasoning substantially increases perceived reasonableness regardless of source, largely by enhancing perceptions of objectivity. Qualitative responses corroborate this tension between objectivity and contextual sensitivity in evaluations of legal advice. Together, these findings suggest that public responses to AI legal advisors are shaped not by rigid attitudes toward automation, but by the balancing of competing normative expectations. The results have implications for theories of algorithm aversion and the design of AI recommendation systems in normatively salient domains.

B. Chen, Zhiyu Li · 0 citations
Open access Aug 2026

Human realignment

Recent advances in AI make it conceivable to delegate legal decision-making to machines, or to enhance human adjudication through AI assistance. Using classic normative conflicts — the trolley problem and comparable moral dilemmas — as a proof of concept, we examine the alignment between AI legal reasoning and human judgment. In our baseline experiment, we find a pronounced mismatch between decisions made by GPT and those of human subjects. This misalignment raises substantive concerns for AI-powered legal decision-aids. We investigate whether explicit normative guidance can address this misalignment, with mixed results. is susceptible to such intervention, but frequently refuses to decide when faced with a moral dilemma. is outright utilitarian, and essentially ignores the instruction to decide on deontological grounds. faithfully implements this instruction, but is unwilling to balance deontological and utilitarian concerns if instructed to do so. We replicate the experiment with four LLMs from different providers. comes closest to human respondents. is most sensitive to normative instructions. and have a strong utilitarian bias, and do not strongly respond to normative interventions. At least for the time being, explicit normative instructions are not fully able to realign AI advice with the normative convictions of the population, or the legislator deciding on its behalf.

Christoph Engel, Yoan Hermstrüwer, Alison Kim · 0 citations
Review May 2026

Assistant or Actor? Student Trust, Control, and Delegation Regret When Using a General-Purpose AI Agent

When AI agents shift from answering questions to taking actions, users face a new problem: deciding what to delegate, to a system whose action space they cannot fully anticipate. We call the resulting dissatisfaction delegation regret, a pattern in which users regret not that the agent erred, but that it acted beyond what they would have authorized. In a controlled study, 20 university students completed five common daily tasks using OpenClaw, a general-purpose AI agent, across tasks chosen to vary in privacy, stakes, and reversibility. For each task we measured trust, perceived control, transparency, supervision burden, and approval preference on 5-point Likert scales, and collected free-text reflections analyzed through thematic coding. Three findings emerged. First, participants calibrated trust per task rather than per agent: they granted wide autonomy for advisory and low-stakes tasks but demanded confirmation for irreversible, externally visible actions. Second, irreversibility combined with external visibility, rather than stakes alone, appeared to drive trust withdrawal: the moderate-stakes email task triggered the sharpest drop in trust (M = 3.10) and the highest demand for approval (M = 4.65), whereas a high-stakes but verifiable task did not produce the same response. Third, delegation regret appeared consistently when the agent executed actions without preview, even when the output was rated as successful. We discuss implications for agent designs that expose action boundaries, support per-task autonomy policies, and separate advisory output from agentic execution.

Shiva Pochampally, Shengwei An, Yan Chen · 0 citations
Preprint Apr 2026

When Direct Prediction Fails: Evidence from LLM-Based Misinformation Risk Evaluation

LLMs make it increasingly easy to generate deceptive content at scale, creating a need for scalable misinformation risk evaluation based on whether readers find such content credible and are willing to share it. A natural approach is to ask an LLM these questions directly and treat the returned scores as predictions of the corresponding human ratings. Implicit in this practice is the assumption that asking about a reader response produces the score that best predicts it. We test this assumption using matched credibility and willingness-to-share ratings for 290 deceptive articles from 317 participants and eight LLM evaluators. Unexpectedly, the assumption holds for credibility but fails for sharing. For every evaluator, credibility scores track human sharing at least as closely as sharing scores, while sharing scores offer no detectable benefit beyond credibility when predicting responses to unseen scenarios. This pattern persists when the questions are asked separately or in reversed order. Our results suggest that directly asking for the target response may not always yield the most effective score for predicting it. Comparing direct scores with indirect paths through related judgments may reveal a more effective predictive route. Deciding what to ask may be as important as refining how to ask it.

Zonghuan Xu, Xiang Zheng, Yutao Wu et al. · 0 citations