Token by token

Over the past few days, I’ve found myself explaining multiple times how large language models (generative AI) work. So, I thought I’d repeat it here for reference.

An LLM doesn’t actually “write an answer”; at least not the way you or I do. It produces a single token at a time (a token can be a word or part of a word) and has no memory of having done so.

Every token is produced by re-reading absolutely everything from the very start of the conversation, both your words and its words, then guessing the most plausible next token. It then repeats this loop: re-read everything, predict one token, append it to the end, repeat. You can see why these things tend to be resource hungry.

That’s the whole scheme: no planning, no long-term memory, no goals, no “intelligence.” Just a very advanced pattern-matching system that keeps asking itself the same question in a loop: “given all the text so far, what word (or part of a word) is most likely to come next?”.

Most of the improvements we get these days are in what’s referred to as harnesses, the software scaffolding that sends and receives tokens in a loop to and from the LLM (like Claude Code or CoWork). And none of that is “AI”; it’s good old-fashioned code. Very impressive code, but still simply code doing what code has always done: following instructions really fast.

Colin

PS: Yes, I massively simplified this on purpose. No "vectors" or "transformers" or “matrix multiplication”; you can go and study linear algebra on your own time.