Arguing with the Internet
Recently, I’ve had several conversations where people come to me with ready-made technical solutions that simply can’t work. Or they send me (wrong) explanations in that unmistakable chatbot voice, but presented as their own thinking.
In all cases, these solutions are delivered with absolute conviction. When I push back, they push back harder.
The source? Generative AI of course. It seems that when the chatbot has spoken, it can’t be wrong.
Well, it’s got a name now. Researchers at Wharton have just published a paper introducing it as “Cognitive Surrender” (great name for a band).
It’s what happens when people stop reasoning and just adopt AI answers as their own, often without even noticing.
In the experiments, whether the AI was right or wrong, the participants’ confidence went up. The AI made people more certain regardless of whether it made them correct.
When someone’s unsure, you can usually tell. And those signals let you know you should push back. Chatbots are confident whatever they’re saying, and this seems to transfer to the people using them.
I’ve written in previous emails how Large Language Models are just text-synthesis engines. They produce plausible sounding output, and factual accuracy is just a side effect of how often the truth appears in their training data.
“Plausible” and “correct” are easy to confuse, especially because our brains associate language use with intelligence. Software that “extrudes” text basically hacks our senses.
The consequence of all this is that you can end up arguing with, not a colleague or a client, but with a statistical echo of internet content that’s delivered with absolute confidence.
The researchers did find one thing that helped: giving people incentives for accuracy. It made them more likely to question the answers, push back, and override. In other words, people need to care about being right, not just about sounding right.
It’s worth thinking about how you use these tools but also how your team does. Before you end up outsourcing your judgement.
Colin
PS: here’s the paper if you’re interested.