Automation bias: Automation bias is the human tendency to trust a suggestion from an automated system over one's own judgment or contradicting evidence, even when the system is wrong. With AI, it shows up as accepting fluent, confident output without checking it.
Human-factors researchers, including Kathleen Mosier and Linda Skitka, documented automation bias in the 1990s while studying pilots and operators working with automated cockpit and monitoring systems. Two patterns kept appearing. People followed automated advice that was wrong (errors of commission), and people failed to act when the system stayed silent, even when other cues pointed to a problem (errors of omission).
Generative AI makes both patterns easier to fall into. Its output reads like a competent colleague wrote it: well organized, grammatical, and delivered without hesitation. None of that is evidence of correctness. A model that invents a case citation formats it exactly like a real one. The polish that makes AI output pleasant to read is the same polish that makes its errors easy to wave through.
The bias grows with fatigue, time pressure, and a track record of the tool being right. That last one is the trap: after a hundred good answers, the hundred-and-first gets less scrutiny, not more. Countermeasures are procedural. Form your own view before reading the AI's on anything that matters. Require a named source for factual claims. Keep sampling outputs even when the tool seems reliable, and track the errors you find so the team has a real sense of how often it fails.
Example at work
A loan processor uses an AI tool that summarizes borrower documents and flags issues. For months the flags are accurate, so she starts approving files with no flags after a quick skim. A missed income discrepancy surfaces in audit. The fix is a rule that every file gets one independent check of income figures against the source document, flag or no flag.
Why it matters
AI tools are increasingly placed in workflows where a person is supposed to be the safety check. Automation bias is what turns that person into a rubber stamp. Knowing the bias exists, and building small rituals to counter it, is what makes human review real.
Related terms
- Human-in-the-loopHuman-in-the-loop is a design approach in which a person reviews, approves, or corrects an AI system's output at defined points before it takes effect, combining the speed of automation with human judgment and accountability.
- HallucinationA hallucination is when an AI model produces information that is false or fabricated but presents it confidently, such as an invented statistic, a citation to a paper that does not exist, or a made-up product feature.
- AnthropomorphismAnthropomorphism is the habit of attributing human qualities such as understanding, intent, or feelings to something that is not human. With AI chatbots, it means reading a fluent, first-person reply as a sign of a mind behind it.
- AI literacyAI literacy is the practical understanding of what AI systems can and cannot do, how they produce their outputs, where they fail, and how to use them responsibly and effectively in your own work.
- ExplainabilityExplainability is the degree to which you can understand why an AI system produced a particular output. It matters most when a decision affects a person and someone has to justify it.