A new study has revealed a troubling paradox: the more people rely on artificial intelligence for answers, the more confident they become in their own judgment—even when the AI is consistently wrong. Research led by Valerio Capraro, a psychology professor at the University of Milan-Bicocca, shows that incorrect AI advice not only reduces accuracy but also inflates users’ confidence in their answers, making them less willing to admit uncertainty.
The study, detailed in a recent report via IBM Think, involved participants answering six obscure questions about movie details—questions deliberately chosen because an AI model answered them incorrectly every time. This allowed researchers to isolate the effect of AI consultation independent of advice quality. The results were striking: without AI, participants declined to answer roughly 36 to 44% of the time across two experiments. With AI available, that figure dropped to just 3 to 6%. Accuracy also plummeted, from 27.5% correct when working alone to only 9.2% with AI involved.
Capraro suspects the AI lowers the threshold of certainty people need before committing to an answer, essentially recalibrating how they judge their own knowledge. Interestingly, the effect persisted even when AI advice appeared automatically without being actively requested, suggesting that passive exposure to AI output can also skew judgment. This challenges the assumption that the danger only arises when people actively seek help—the mere presence of AI suggestions may be enough to override natural caution.
Why does wrong AI advice make people so sure of themselves?
The phenomenon ties into cognitive biases like overconfidence and automation bias, where humans tend to trust machine-generated recommendations even when they contradict available evidence. In the study, participants who received incorrect AI advice became noticeably more certain of their own answers, despite being demonstrably less accurate. They also became far less willing to admit they didn’t know an answer, even when the option to remain silent was clearly available and they were rewarded for accuracy.
The research mirrors earlier findings on “meta-cognitive errors,” where external aids can distort self-assessment. For decades, psychologists have known that providing people with answers—even wrong ones—can shift their internal calibration of confidence. What’s new here is the scale and immediacy of AI’s influence. Unlike a human adviser or a textbook, AI systems often deliver answers with an aura of precision and authority, which may amplify the effect. Capraro’s team also observed that the confidence boost was not limited to people who explicitly trusted the AI; even those who were skeptical showed similar patterns, indicating a subconscious influence.
Another factor may be the speed and ease of AI interaction. When advice comes instantly and without effort, it may bypass the usual mental checks that prompt doubt. The study controlled for this by making the AI advice appear automatically in one experiment, and the results remained unchanged—suggesting that the key driver is not the act of seeking help, but the mere presence of an apparent “expert” answer.
Does raising the stakes change anything?
To test whether consequences could counter the effect, the researchers introduced financial rewards and penalties for correct and incorrect answers. They found that this made people somewhat more cautious and accurate, but the underlying pattern never fully disappeared. Even with money on the line, participants exposed to incorrect AI advice still displayed higher confidence and lower accuracy than those working alone. This indicates that while incentives can mitigate the effect, they cannot eliminate it entirely.
Capraro draws a clear distinction: AI should augment human judgment, not replace it entirely. He worries this effect extends to children, who are increasingly growing up with instant answers from voice assistants and chatbots. If young minds never experience the productive discomfort of doubt during learning, they may fail to develop the critical thinking skills needed to evaluate information. A wave of recent reports suggests that the AI-addicted younger generation is already turning to chatbots for help with everything from homework to in-person conversations.
“Doubt isn’t a failure of knowledge,” Capraro said. “It’s often where real knowledge actually begins.” His comment underscores a broader concern in educational psychology: that the ease of AI-driven answers may short-circuit the very process of inquiry that builds deep understanding. Studies on “cognitive offloading” have shown that when people rely on external tools (like calculators, search engines, or now AI) to answer questions, they retain less information and become less adept at reasoning from first principles.
The implications extend beyond education. In professional settings—from medicine to finance to journalism—the same dynamics could lead to overconfident decision-making when AI advice is flawed. If a doctor uses an AI diagnostic tool that makes subtle errors, the research suggests they may become more convinced of their own assessment while ignoring warning signs. Similarly, financial analysts might overcommit to strategies validated by an AI model that is actually biased.
Capraro’s study is part of a growing body of work examining the psychological impact of AI interaction. Earlier this year, researchers at Stanford found that people who used AI writing assistants became more confident in their own writing ability, even when the AI introduced errors. Another study from Harvard noted that reliance on AI for medical diagnoses reduced doctors’ willingness to second-guess their initial impressions. Together, these findings paint a picture of a technology that, while enormously powerful, can subtly undermine human judgment if not used with caution.
The open question is how best to design AI interfaces that encourage critical thinking rather than blind acceptance. Some experts advocate for “explainable AI” that highlights uncertainty and prompts users to reflect before accepting advice. Others recommend training programs that teach people to recognize their own cognitive biases when interacting with AI. Still, the new research suggests that the effect may operate below conscious awareness, making it difficult to counteract through awareness alone.
For now, Capraro’s work serves as a cautionary tale. The very tool designed to augment human intelligence may, under the wrong conditions, do the opposite—not by replacing thought, but by disabling the healthy skepticism that prompts us to think twice. As AI becomes more integrated into daily life, understanding when and why it breeds false confidence will be crucial for ensuring that we remain the masters of our own decisions.
Source: Digital Trends News