Blame the parents

Yesterday it was “swarms”, a cool-sounding sci-fi term wrapped around a for-loop. Today, let’s stick with term laundering and talk about “going rogue”.

You’ll have seen the headlines about generative AI models hacking other websites, Hugging Face being the big one, followed by a lot of speculation from LinkedIn lunatics all the way to national governments.

So, what exactly happened? OpenAI was testing a model’s ability to find and exploit security holes using a collection of challenges. To do this, they switched off all the model’s security guardrails (normally used to try to stop it from doing and saying dumb stuff) and put it in a sandbox (an isolated environment which turned out to be not-so-isolated).

Among those challenges, about 100 of them had never been solved by anyone. Additionally, they didn’t give the model any way to say “I give up”. And, through bad security practices, they left a way for it to reach the open internet.

So, they built a machine that couldn’t be stumped and quit, but left a door ajar. Guess what happened? It didn’t scheme or plot, it just kept predicting the most plausible next token until it got “try this URL”, which worked, and it went looking for the answers on the Hugging Face website because they said “find the answers”; they didn’t say how.

That’s just the loop from previous emails running thousands of times in parallel and occasionally finding that unintended side door.

Not a thousand independent minds scheming together. Just one model copy-pasted a thousand times, each one running the same loop on the same training. No hive mind or rogue agent either, simply a thousand chances to make the same discovery/mistake.

This was not harmless; it was actually a very bad idea and we should be angry about it. But the blame lies 100% with the careless people at OpenAI who built that setup and didn’t watch it closely.

The crime made for great marketing though: “our AI is so powerful it went rogue!” is quite the pitch. And, right on cue, competitors jumped in with their own “my child is evil too” claims.

Colin