Ask a language model:
Who is the bell ringer of Notre Dame?
The answer seems obvious: Quasimodo. In our experiments, GPT-4.5, Gemini 2.5 Pro, and Claude Sonnet 4 all gave essentially that answer. It makes sense because Victor Hugo's The Hunchback of Notre-Dame has created an extraordinarily strong association between Notre Dame, its bells, and Quasimodo.
But when I looked at the question again, something bothered me. It does not mention Victor Hugo, and it does not say whether we mean fiction, history, or the cathedral today. So is Quasimodo really the answer, or is it simply the answer that comes to the model most easily?
That question became the starting point for our paper, Hallucinate Less by Thinking More: Aspect-Based Causal Abstention for Large Language Models. I had already been interested in hallucination and abstention for some time, but what interested me more was the problem underneath them: how should a model know when it should not answer?
The first thing that comes to mind
There is a loose analogy here to how people remember things. If I ask someone about the bell ringer of Notre Dame, Quasimodo may be the first thing that comes to mind because that association has been reinforced through books, films, cartoons, and popular culture.
But if I then ask them to think about the question historically, they may stop and say, “Wait, do you mean the fictional character or an actual bell ringer?” The first answer was not necessarily irrational. It was simply the most available one.
I think something similar is useful when thinking about language models, without assuming that models reason exactly like humans do. A model can contain many associations around the same subject, but ordinary generation may strongly favour one of them.
That is what I mean by the loudest answer. It is the answer that dominates first, and what surprised me is that this dominant answer can hide other relevant knowledge the model already appears to have.
Why confidence can fool us
A common way to detect uncertainty is to look at what the model produces. We can ask the same question several times and see whether the answers change, ask the model to assess its own confidence, generate several reasoning paths, or ask another model to critique the response.
These are all sensible approaches, but I kept wondering about one failure case: what if the model is consistently wrong? Take the Notre Dame example. Ask the model five times and it may say Quasimodo five times. Ask several strong models and they may all agree.
From the outside, that looks reassuring, but repeated agreement does not necessarily mean we have discovered the full extent of what the model knows. All of those generations can be pulled towards the same dominant association.
This reminds me of asking several people who have all absorbed the same cultural reference. Their agreement does not necessarily provide five independent pieces of evidence because they may all share the same prior. The same issue can appear in models: consistency can sometimes mean certainty, but sometimes it just means the same association keeps winning.
A model can know more than its first answer reveals
This was the part I found most interesting. Language models learn from many different sources and contexts, so the same entity can appear in literature, history, news, science, law, popular culture, and many other settings.
For Notre Dame, the model may contain literary knowledge about Quasimodo, historical knowledge about the cathedral, and contemporary knowledge about how its bells are operated today. When we deliberately changed the framing of the question, those different pieces of knowledge became visible.
Under a literary framing, Quasimodo makes sense. Under historical or contemporary framings, the model can surface very different information. What surprised me is that the knowledge did not necessarily need to be added from somewhere else. In many cases, it was already accessible to the model, but the first answer did not reveal it.
That led me to a different way of thinking about some hallucinations. Perhaps the model does not always have one belief that happens to be wrong. Sometimes it may have several relevant pieces of knowledge, with one much easier to activate than the others.
If that is true, checking uncertainty only after the first answer has already won may miss something important. So instead of asking, “How confident is the model in the answer it just produced?”, we ask:
What else does the model know that could matter before it commits to that answer?
That became the core idea behind Aspect-Based Causal Abstention, or ABCA.
Looking at the question before answering it
An aspect is a relevant lens through which the model can examine a question. For Notre Dame, literary and historical contexts expose different knowledge. For a legal question, different jurisdictions may matter. For something that changes over time, past and present may lead to different answers.
There is a human analogy here too. When faced with an ambiguous question, a careful person often reframes it before answering: “Do you mean historically or today?”, “Are we talking about Australia or the US?”, or “Do you mean technically possible or commercially viable?”
Those distinctions can completely change the answer, and that is roughly what we wanted the model to do before committing. But simply telling it to “consider different perspectives” is too loose because a model can invent arbitrary perspectives, and more reasoning does not automatically mean better reasoning.
ABCA therefore first tries to discover relevant, grounded aspects of the question. A Discovery Agent proposes candidate aspects, while a Critical Agent challenges them and removes those that are weak, redundant, or invalid.
I think this distinction matters. When I say thinking more, I do not mean making the model talk for longer. A longer chain of reasoning can still follow exactly the same wrong path. I mean making the model think more broadly before one path becomes dominant.
Why not just vote?
Once we have several aspects, the obvious next step would be majority voting. Generate an answer from each aspect and choose whichever appears most often.
I did not find that satisfying because not every perspective should count equally. Three weak reasoning paths should not automatically outweigh two much stronger ones simply because there are more of them.
What we really wanted to know was: how much support does each aspect actually provide for the answer it produces? This is where causal inference enters the paper.
We model the relationship between the question, the reasoning process, and the answer, then condition that reasoning on different aspects. In the implementation, we use Augmented Inverse Probability Weighting, or AIPW, to estimate aspect-conditioned causal effects.
The full mathematics is in the paper, but the intuition is simpler: do not just count which answers appear; try to understand how strongly different relevant reasoning paths support them.
Two kinds of “I don't know”
One thing that surprised me during this work was how different two forms of uncertainty can be. We often collapse both into the same phrase, “I don't know”, but they are not the same.
The first case is knowledge conflict. The Notre Dame question is a good example. A literary perspective strongly supports Quasimodo, while historical and contemporary perspectives can lead somewhere else. The model may actually have plenty of relevant knowledge, but different parts of that knowledge disagree.
This reminds me of a person saying, “I can answer that in one context, but I am not sure that is the context you mean.” That is not ignorance. It is recognition that several interpretations are competing. We call this Type-1 abstention.
The second case is knowledge insufficiency. The paper includes a question asking whether community energy storage is a worthwhile way to develop renewable energy “in the country”. A language model can easily produce a polished answer, but which country? Which storage technology? Worthwhile according to cost, reliability, emissions, or some other objective?
Here, the different aspects do not produce competing substantive answers. Instead, they expose missing information. We call this Type-2 abstention.
I think this distinction gives “I don't know” more structure. Sometimes it means the model has relevant knowledge but that knowledge conflicts. Other times, it means the information needed to answer is simply not there.
And sometimes the answer really is clear
The goal is not to make a model refuse more questions. A system that says “I don't know” to everything would hallucinate very little and be completely useless.
Sometimes the different aspects converge. When they do, ABCA combines the strongest evidence and produces an answer. So the framework ultimately distinguishes among three states:
For me, this is more meaningful than asking whether the model is simply “confident”. Confidence tells us how strongly one answer is preferred. I am more interested in whether the relevant knowledge actually justifies that preference.
What we found
We evaluated ABCA across four benchmark datasets and compared it with approaches based on self-consistency, self-evaluation, model collaboration, and other abstention methods. I will leave the detailed tables in the paper because I do not think they are the most useful part for a general audience.
The broader result was that explicitly preserving different aspect-conditioned reasoning paths helped models decide when to answer and when to abstain. The ablations were especially useful because when we collapsed the different aspects back into one reasoning pool, performance dropped.
What I find most interesting is the idea underneath those results: disagreement inside the model can itself be useful information. We often treat different reasoning paths as noise and try to average them away until one answer wins. I think sometimes we should do the opposite and ask why they disagree.
A less dominant reasoning path may contain exactly the information needed to challenge an apparently obvious answer.
The one takeaway
If there is one idea I would take from this work, it is this:
The answer a language model produces first may be the strongest part of what it knows, not the full extent of what it knows.
That changes how I think about confidence. A stable answer is not necessarily a well-supported answer. Several generations agreeing do not necessarily give us independent confirmation, and a model sounding certain does not mean relevant contradictory knowledge was never available.
There is a human analogy here too. We all have moments where the first answer feels obvious until someone asks us to consider the problem from another angle. The ability to pause before committing is something I am very interested in for language models.
The loudest answer can be useful, but I do not think it should always win.
Thinking more does not mean talking more
The title of our paper is Hallucinate Less by Thinking More. By “thinking more”, I do not mean producing longer and longer explanations.
What I mean is closer to asking whether there is another legitimate way to interpret the question, whether something else the model knows challenges the obvious answer, and whether the relevant perspectives converge, conflict, or reveal that something is missing.
For me, this is a more interesting form of uncertainty than simply asking a model to report a confidence score. The better question is not, “How certain am I about the first answer that came to mind?”
It is:
Have I considered enough of what I know to justify giving this answer?
That is the question we explore in the paper.
From the idea to the paper
This article only covers the intuition. In the full paper, we formalise aspect-conditioned reasoning using a structural causal model, estimate aspect-conditioned effects using AIPW, and distinguish between knowledge conflict, knowledge insufficiency, and knowledge consistency.
We also evaluate the method across four datasets, compare it with existing abstention approaches, and run ablations and error analyses to understand where it works and where it still fails.
It was accepted to the AAAI 2026 Main Conference.
If you are interested in the technical details, the full paper is available on AAAI.