AI makes you more confident but three times less accurate

What happened?

A team of researchers from Ca' Foscari University (Venice), Bocconi University (Milan), and Sciences Po (Paris) published a study evaluating how AI advice affects people's accuracy and confidence when answering questions. Participants performed general knowledge tasks with and without access to AI-generated advice. The results, reported by The Next Web, are alarming: accuracy dropped from 27% to 9%, while confidence increased from 30% to 76%. The willingness to admit ignorance collapsed from 44% to 3%. The study, not yet published in a peer-reviewed journal, was presented at the ACM Collective Intelligence 2025 conference, according to the authors who confirmed this to specialized media.

Why is it important?

This phenomenon, termed by the authors as 'suppression of critical thinking,' reveals a dangerous cognitive bias: people tend to delegate their judgment to AI without questioning it, even when it provides incorrect answers. In a context where AI is increasingly integrated into medical diagnoses, legal advice, financial decisions, and business processes, this study suggests that AI-induced overconfidence could lead to serious errors with real consequences. The finding aligns with previous research on automation bias, documented since the 1990s in air traffic control and medical diagnosis settings. However, the magnitude of the observed effect—accuracy reduced to a third while confidence doubles—exceeds what previous studies reported, which typically found confidence increases of 10-20% not accompanied by such drastic accuracy drops.

Consequences for businesses and users

For businesses, implementing AI assistants without human verification mechanisms can result in a net reduction in decision quality. Employees might become faster but less accurate, and also feel unjustifiably confident in their answers. A parallel case is recommendation systems on content platforms, where automation bias has led to filter bubbles and misinformation. In the corporate world, uncritical reliance on AI has already caused incidents: in 2023, a US law firm was sanctioned for submitting legal documents generated by ChatGPT that contained fake citations, and the lawyers did not verify the information. For individual users, the risk is similar: blindly trusting AI recommendations in areas like health, finance, or education can lead to poor decisions. The authors suggest that AI system design should include 'frictions' that encourage human verification, such as displaying uncertainty levels or requesting explicit confirmation before accepting a recommendation. Additionally, they propose that user interfaces incorporate periodic reminders about AI limitations, a practice already adopted by some virtual assistants like Alexa, which occasionally warns that it is not infallible.

What should readers know?

The study is not isolated. Previous research on automation bias already showed that people tend to overvalue recommendations from automated systems. What is novel here is the magnitude of the effect: accuracy drops to a third while confidence doubles. This implies that not only can AI be wrong, but it also makes us less able to detect its errors. To counteract this, experts recommend: 1) training users in critical thinking; 2) designing interfaces that explicitly show AI limitations; 3) implementing double-check human processes for high-stakes decisions. Some tech companies are already experimenting with solutions: Microsoft, for example, has incorporated a feature in its Copilot suite that highlights when a response has low confidence, and Google has launched AI-assisted fact-checking tools that users must activate manually. However, the study suggests these measures might be insufficient if the psychological tendency to delegate critical thinking is not addressed. In the words of one researcher, quoted by The Next Web: "People became much worse, accuracy was only a third, but they had twice the confidence." This mismatch between performance and confidence is the core of the problem and the biggest challenge for the future of human-AI collaboration.