Verification immunity

Yesterday, writing about the invulnerability bias, I touched on the related idea that using generative AI to enhance work usually lacked the appropriate guardrails.

The most obvious guardrail is simply checking the output.

Interestingly, nearly every time I talk with someone who uses generative AI in their work, and mention the non-deterministic (not always the same result from the same input) aspect of these chatbots, they insist that they always check the work.

To put it bluntly: I don’t believe them. And, I say that as someone who’s been there.

There’s another bias at work here: the automation bias. It’s the tendency to trust automated systems above your own judgement. It can be found in all kinds of places: from GPS sending people into lakes, to the UK Post Office Horizon scandal.

Added to that, we have verification complexity. When verification is cognitively demanding, people default to trusting the system – even if they honestly intended to check initially.

Combine those two and you get this pattern: people verify once or twice in the beginning, then the cognitive costs of the repeating work lead them to complacency.

We see this every day: lawyers citing non-existent cases in court, magazine articles recommending books that don’t exist, travel guides with restaurants that were never there, the list goes on…

A bad reference in an email might not cause much damage. But a generated spreadsheet you didn’t check, a dashboard with the wrong numbers (I saw this just recently), or a report confidently stating something that’s not true; those might cost you credibility, clients, or even your job.

Expecting anyone, myself included, to verify everything isn’t realistic.

Colin