When They Err, They Don't Err the Same Way

We ended the previous part with a question: if the model isn't after truth, what does it do when it doesn't know something? It certainly doesn't stay silent. To answer that, we have to see that the errors of these two systems are not of the same kind.

Diagnostic software is not error-free. Sometimes it misses a lesion that is there; sometimes it reads a shadow as a lesion. But pay attention to the nature of this error: even when it reads a shadow as caries, that shadow really is on the radiograph. Something real was there that misled it. Its error is the misreading of an existing thing, and you can trace it back to the very spot that threw it off.

The language model is different. When it hasn't seen something often enough across those vast bodies of text, it does not pause at the fork between “I don't know” and “make something up”; for it, no such fork even exists. Its job from the start has been to place the most probable next word, and that is exactly what it does — whether or not there is any reality behind it. It lines words up to build something that has the shape of a correct answer.

And here is the essential difference. The error of diagnostic software is always tied to something real; something was there and it misread it. But the error of a language model can be tied to nothing at all. There was no article it misread — the article never existed in the first place. The one mislabels a real spot; the other can place before you, with a name and details, something that exists nowhere.

And this is where the danger grows. The software's error can be caught, because the radiograph is right in front of you and you weigh your own reading against its reading. But when the model builds something that doesn't exist, there is nothing to compare it against; only a fluent, confident text that has the shape of a real answer. The only thing that gives it away is your own knowledge. If you don't have that knowledge, there is no sign in the text itself to tell you that it made this piece up.

This error — the model, with complete confidence, building something that has no external existence — is so important that it has a name of its own. We will devote the entire next part to it. First, we have to see it, face to face.