Even the Pentagon fell for it

Remember automation bias and verification complexity? I’ve mentioned them previously as the reasons the majority of users never check generative AI output.

This spring, US armed personnel were about to board a Chinese ship in the Middle East reportedly carrying nuclear weapon components. At the last minute, officials took a closer look and realised the report had been “hallucinated” by an AI.

Some special ops analyst asked the military chatbot “what’s on this ship?” and it said “nuclear components”. He then threw that back at the AI, asked it to generate a report, and sent it up the chain. Next thing you know, the US was on the brink of war with China.

Any time you mention that these systems confabulate, you always get the same soothing replies: don’t worry, there’s a human in the loop.

Well, there was one here. He prompted twice and hit send.

If the Pentagon gets caught out by these mechanisms, how do you think other organisations like insurers or local councils or… yours will fare?

“Human in the loop” sounds good. But if that human takes even one shortcut, you don’t have a human in the loop. You have one standing next to it.

Source: CNN

Colin