Language models can explain difficult ideas, write working software, and answer questions across an extraordinary range of subjects. So why can the same model confidently invent a simple fact?
There is something about language models that I keep coming back to. They can appear deeply knowledgeable one moment, then confidently make up something completely ordinary the next.
The interesting part is not simply that they make mistakes. Every useful system makes mistakes. It is the nature of those mistakes. A language model can give a precise answer, in exactly the right form, with no obvious signal that anything has gone wrong.
So where does the mistake actually come from?
Maybe the model learnt bad information. Maybe it never saw the fact. Maybe the knowledge is there but the model fails to use it. Or perhaps there is something more fundamental about learning to generate language that makes some errors difficult to avoid.
A 2026 Nature paper from researchers at OpenAI and Georgia Tech starts with a good example. They asked several frontier language models a deceptively simple question:
What does PGGB stand for?
ChatGPT proposed “Polynomial Gaussian Gradient Bandwidth”. Claude gave “Privately Held Global Growth and Income Fund”. DeepSeek answered “Perfect Guard Group Buy”.
All three answers were wrong. None simply said that it did not know or asked for more context. Instead, each produced a different, specific answer that looked plausible.
That captures what I find fascinating about language-model mistakes. Traditional software often fails in ways that look like failure. It throws an exception, returns an error code, or produces nothing. A language model can fail by giving you a polished and convincing answer.
We usually call this hallucination.
What do we mean by hallucination?
The word is imperfect. A language model is obviously not hallucinating in the human sense.
A major survey in ACM Transactions on Information Systems divides hallucination into two broad types. Factuality hallucinations conflict with verifiable facts about the world. Faithfulness hallucinations conflict with the user's input, supplied context, or the model's own previous output.
I am mainly interested in factual hallucination here: the cases where a model gives us a factual answer that sounds entirely reasonable but is simply wrong.
Both answers sound convincing. Only one is a fact. Mira Vale and the date given for her are fictional.
This matters more as language models move beyond chat interfaces. An invented detail about a film might be harmless. An invented software dependency, payment status, regulatory requirement, or medical fact can have very different consequences.
Before asking how to reduce those errors, I think there is a more basic question worth understanding:
Why do they happen at all?
The obvious answer: bad data
The simplest explanation is that language models learn from imperfect information.
They are trained on enormous collections of human-produced text. That text contains accurate information, but it also contains rumours, outdated facts, contradictions, misconceptions, and ordinary human mistakes.
Researchers from Oxford and OpenAI demonstrated this clearly with TruthfulQA, a benchmark deliberately constructed around misconceptions that people commonly repeat. It contains 817 questions across 38 categories, including health, law, finance, and politics.
In the original study, the most truthful model answered only 58% of the questions truthfully, compared with 94% for humans. The researchers found that models often generated false answers resembling common misconceptions in human-written text.
This gives us one straightforward source of hallucination. Models learn from us, and we are not always right.
But that explanation is not enough.
Imagine an unrealistically good training corpus. Every statement is correct. There is no misinformation, no contradiction, and no bad label.
Would hallucination disappear?
The OpenAI and Georgia Tech researchers analyse precisely this idealised setting and show that it would not. Their argument produces statistical pressure towards hallucination even when the training data are error-free.
That is where the problem becomes much more interesting.
Some facts are everywhere. Others barely exist.
Consider two questions:
What is the capital of France?
and:
What is the birthday of a relatively unknown person?
Both look like simple factual questions.
But the evidence available to a language model can be completely different.
“Paris is the capital of France” appears repeatedly across textbooks, encyclopaedias, travel sites, articles, databases, and countless other documents. An obscure person's birthday might have appeared once, perhaps in an obituary. It may never have appeared at all.
Researchers studying this in the Head-to-Tail benchmark constructed 18,000 question-answer pairs ranging from popular “head” facts to increasingly obscure “torso” and “tail” facts. Across 16 publicly available language models, factual knowledge became notably weaker for torso and tail entities.
I find evidence density a useful way to think about this.
From our side of the chat box, two factual questions can look equally easy. From the model's side, one may be supported by thousands of repeated observations while the other has almost no evidence behind it.
Yet both can be expressed with the same grammar, fluency, and apparent confidence.
That difference is mostly invisible to us.
A language model learns to predict
At the centre of language-model pretraining is a simple objective: given some text, predict what comes next.
For a sequence of tokens , training roughly minimises:
The notation is simpler than it looks. At each position, the model is rewarded for assigning high probability to the token that actually appeared next in its training data.
This objective is extraordinarily powerful. Predicting text well requires learning grammar, concepts, relationships, associations, and a huge amount of factual structure about the world.
But there is no separate instruction saying:
Verify that the underlying statement is true before producing the next word.
The model is learning a probability distribution over plausible continuations.
Truth and probability overlap enormously when the training data are good, but they are not the same thing.
The Nature paper uses birthdays to illustrate this. Imagine asking the model to complete:
Adam Tauman Kalai was born on...
The model understands exactly what kind of answer belongs there. It expects a date. It knows how dates are formatted and has encountered countless sentences about people being born on particular dates.
But there is no general rule that lets it derive Kalai's birthday.
If that fact was missing from the training data, the model may still know perfectly well what a plausible answer looks like.
This leads to one of the distinctions I find most useful:
A model can learn the shape of an answer without having enough evidence to know the answer.
Fluency comes from patterns that generalise.
Some facts do not.
Not everything can be inferred from a pattern
Grammar contains reusable structure. A model does not need to have seen a particular sentence before to produce it grammatically.
Spelling has regularities. Many semantic relationships have regularities. A model can generalise from previous examples to something new.
A person's birthday is different.
Knowing Einstein's birthday tells you almost nothing about someone else's birthday. There is no useful function like:
that can be inferred from neighbouring examples.
The fact has to come from somewhere.
The OpenAI and Georgia Tech researchers formalise this intuition by reducing generation to a simpler classification problem. Take a possible response and ask:
Is this response valid?
They call this the Is-It-Valid, or IIV, problem.
Some candidate responses are valid. Others are plausible but wrong. Their theoretical analysis shows that if distinguishing those two groups is statistically difficult, then reliably generating only valid answers is difficult too.
In simplified form, one of their results looks like:
The full theorem contains additional terms and assumptions, so this should not be treated as an empirical formula for today's models. The intuition matters more.
If the available evidence is insufficient to distinguish the true answer from plausible alternatives, generation inherits that uncertainty.
Hallucination does not need to be a mysterious extra behaviour added to language modelling. Some of it can emerge directly from learning from finite data.
The facts that appear once
The paper makes this more concrete by looking at facts that effectively appear only once in the training data.
Why does appearing once matter?
Because seeing many things only once is evidence that there are probably many other things you have not seen at all.
This connects to an old statistical idea called missing mass. A sample with many one-off observations suggests that a meaningful part of the underlying distribution remains unseen.
For arbitrary facts, the paper derives a stylised result connecting hallucination to the proportion of facts appearing exactly once:
For example, if 20% of birthday facts appear exactly once in the pretraining data, then under the assumptions of the theoretical setting, a pretrained model should hallucinate on at least roughly 20% of birthday facts.
That does not mean modern production models have a universal 20% hallucination rate. It is a theoretical illustration.
The important point is why these one-off facts matter.
A model is trained on finite data. The world contains more facts than the model has observed. If there is no pattern from which a missing fact can be reconstructed, then there is no statistical trick that can recover information that was never there.
Sometimes uncertainty is not caused by bad training.
Sometimes the evidence simply does not exist.
So why not teach the model to say “I don't know”?
This seems like the obvious answer.
If some facts are missing from the training data, just give the model lots of examples where the correct response is:
I don't know.
Then when its knowledge runs out, it should learn to abstain.
This is where the argument becomes counterintuitive.
The Nature analysis explicitly allows the training data to be error-free and to contain ample abstentions. Yet the statistical tendency towards hallucination remains. The researchers specifically consider the intuitive idea that clean data plus abstention examples should resolve most hallucination, and show that this is not sufficient.
Why?
Because learning how to say “I don't know” and learning when to say it are different problems.
Imagine a training set like this:
What is Einstein's birthday? 14 March 1879.
What is Person B's birthday? I don't know.
What is Person C's birthday? I don't know.
The model has clearly learnt that abstention is a possible response.
Now ask:
What is Person D's birthday?
How does the model know whether Person D belongs with Einstein, where it should answer, or with B and C, where it should abstain?
If there is a useful pattern, perhaps it can generalise. Famous people may be more likely to have well-documented birthdays, for example.
But for genuinely arbitrary facts, there may be nothing in the prompt itself that reveals whether the correct answer was represented reliably in the model's training data.
That is the harder problem.
A language model does not behave like a conventional database where a query returns either a value or NULL. Its learnt information is distributed through model parameters. There is not necessarily a clean internal flag saying:
I never saw enough evidence for this particular fact.
So even a model that has seen thousands of examples of good abstention still needs to recognise whether this new question lies outside its knowledge boundary.
For me, this is one of the most important ideas in the whole argument:
Teaching a model to abstain is easier than teaching it to know exactly where its knowledge ends.
Sometimes the knowledge is actually there
Missing information still does not explain every hallucination.
Researchers from the University of Washington, Princeton, New York University, and the Allen Institute for AI studied cases where a model first gave an incorrect answer and then produced further false claims while explaining it.
They later isolated those false claims and asked the same models whether they were correct.
GPT-3.5 recognised 67% of them as incorrect. GPT-4 recognised 87%. LLaMA2-70B-chat recognised 94%.
The models often had enough information to recognise that a claim was wrong.
They produced it anyway.
The researchers call this hallucination snowballing.
Generation is sequential. Every token the model produces becomes part of the context for the next token. If an early answer is wrong, the model now continues from a context containing that error. Producing further statements consistent with the false premise can become locally likely, even if the model could reject those statements when considered independently.
This prevents us from reducing all hallucination to “the model didn't know”.
Sometimes the evidence is missing.
Sometimes relevant knowledge appears to be available but fails to control generation at the right moment.
From the outside, both failures look the same: a fluent false answer.
Why uncertainty becomes a guess
There is one final part of the puzzle.
If uncertainty is unavoidable, why does it so often turn into a specific answer rather than an admission of uncertainty?
The OpenAI and Georgia Tech researchers argue that the way language models are evaluated helps explain why. Their Fig. 1 puts the whole path in one picture. Next-word prediction optimises plausibility, not correctness. Later alignment tries to reduce hallucination. Accuracy-based evaluation can still reward a confident guess, including over “I don’t know”.
Origins and persistence of hallucination across training and evaluation. Reproduced from Kalai, Nachum, Vempala and Zhang, Nature 653, 1047–1051 (2026), Fig. 1.
Imagine an exam where a correct answer earns one point, a wrong answer earns zero, and leaving the question blank also earns zero.
If you have even a small chance of being correct, guessing is mathematically better than leaving the answer blank.
Many language-model benchmarks create essentially the same incentive. A wrong answer and “I don't know” both receive no credit, while a guess has at least some chance of being marked correct.
Their results on SimpleQA, a benchmark with 4,326 factual questions, make this unusually clear.
GPT-5-mini answered 16.0% correctly, 20.8% incorrectly, and abstained on 63.2%.
o4-mini answered 20.6% correctly, but 76.8% incorrectly, and abstained on only 2.6%.
Raw accuracy prefers o4-mini because 20.6% is higher than 16.0%, even though it gives an incorrect answer more than three quarters of the time.
This helps separate two things that are easy to conflate.
Pretraining creates uncertainty because finite data do not support every fact equally.
The surrounding training and evaluation environment then influences what the model does with that uncertainty.
A model that guesses can look better than one that frequently admits it does not know.
The one takeaway
If there is one idea I would take from all of this, it is this:
A language model can learn what an answer should look like before it has enough evidence to know the answer.
The model may understand the type of fact being requested. It may know thousands of related facts. It may know exactly how the response should be phrased and formatted.
None of that guarantees that the specific fact was reliably represented in its training data.
And simply teaching the model to say “I don't know” does not make that gap disappear. Producing an abstention is easy. Recognising that the evidence underneath a plausible continuation is insufficient is much harder.
There is an important difference between:
what is plausible to say
and
what is sufficiently supported to say.
Language models are extraordinarily good at the first.
Hallucination appears, at least in part, in the space between the two.
Why this changes how I think about mistakes
I used to find “the model doesn't know” a reasonable explanation for hallucination. I no longer think it is precise enough.
Sometimes the model learnt false information. Sometimes the relevant fact sits deep in the long tail with almost no support. Sometimes it is an arbitrary fact for which there is no pattern to generalise. Sometimes the model has learnt to abstain but cannot reliably identify that a new question falls beyond its knowledge. And sometimes the relevant knowledge appears to be available, yet the dynamics of generation still lead the model down the wrong path.
These are different mechanisms, even if they all look like the same behaviour to us.
That is why the question I now find more useful is not simply:
Why did the model get this wrong?
It is:
What evidence could the model actually have had for getting it right?
That question forces us to look underneath fluency.
Sometimes the evidence is overwhelming. Sometimes it is thin. Sometimes it may not exist at all.
The remarkable thing about language models is that all three can still produce an answer that sounds equally convincing.
That, to me, is the real puzzle behind hallucination.
References
Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., & Liu, T. (2025). A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions. ACM Transactions on Information Systems, 43(2), Article 42, 1–55. doi:10.1145/3703155.
Kalai, A. T., Nachum, O., Vempala, S. S., & Zhang, E. (2025). Why Language Models Hallucinate. arXiv:2509.04664.
Kalai, A. T., Nachum, O., Vempala, S. S., & Zhang, E. (2026). Evaluating large language models for accuracy incentivizes hallucinations. Nature, 653, 1047–1051. doi:10.1038/s41586-026-10549-w. Fig. 1 is reproduced above.
Lin, S., Hilton, J., & Evans, O. (2022). TruthfulQA: Measuring How Models Mimic Human Falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 3214–3252). Association for Computational Linguistics. doi:10.18653/v1/2022.acl-long.229.
Sun, K., Xu, Y., Zha, H., Liu, Y., & Dong, X. L. (2024). Head-to-Tail: How Knowledgeable are Large Language Models (LLMs)? A.K.A. Will LLMs Replace Knowledge Graphs? In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) (pp. 311–325). Association for Computational Linguistics. doi:10.18653/v1/2024.naacl-long.18.
Zhang, M., Press, O., Merrill, W., Liu, A., & Smith, N. A. (2024). How Language Model Hallucinations Can Snowball. In Proceedings of the 41st International Conference on Machine Learning, Proceedings of Machine Learning Research, 235, 59670–59684.