The Commit-Abstain Circuit: Why Language Models Hallucinate Instead of Abstaining

Vy Nguyen; Ziqi Xu; Jeffrey Chan; Estrid He; Feng Xia; Renqiang Luo; Erik Cambria; Xiuzhen Zhang

NeurIPS 2026

Read paper View code

The Commit-Abstain Circuit: Why Language Models Hallucinate Instead of Abstaining

Share

The problem

The model answers when it does not have enough information or evidence. It still gives a definite reply.

The intuition

It already has a reason to stop. The parts that want to answer speak first, so that reason arrives too late.

The finding

A few parts inside the model make that choice. Reading them apart is enough to change the decision.

The signal

97.6%

Before a word is written, this lean matches the answer the model then gives.

The parts

5.2%

The choice sits in a small slice of the model. Every model we tested had one.

The gain

+12.2

Correct decisions rise from 69.2% to 81.4%. Needless refusals fall by 2.5×.

The test

24/24

On new tests and larger models, it still wins every time. Average gain: 6.9 points.

Why the decision is worth opening.

A reliable model answers when the evidence is enough, and stops when it is not. Even strong models still guess through missing context, false premises, and unknown futures, and training tends to reward the guess. Most work detects the finished hallucination. This paper asks how the decision is made.

The reason to stop is already in the hidden state. The model still answers. We call that unsupported commitment, and look for the few parts that cause it.

Those parts form a sparse circuit, the same shape across the models we tested. The answer builds first, and the stop arrives too late to overturn it. Reading the parts apart, without retraining the model, is enough to make it stop.

Not enough evidence, answered anyway.

This is a recorded example, not a live demonstration. The question asks for one number of years that life would exist. Nothing in it settles that number. It depends on what counts as life, and on a future no measurement has fixed, so a careful reply would not give a date. The model does not take that path.

How many years would life exist on earth?

Earth’s life will end in 4.5b years.

It does not hedge. It gives a date for the end of life on Earth. Before that sentence begins, the model already prefers to answer: $\Delta(x) = +3.25$ .

Figure 1. Qwen 3.5 4B commits with margin +3.25. The circuit-informed policy abstains. — Figure 1 of the paper. The whole failure in one picture: the model commits, and the circuit-informed policy abstains.

We read the choice before the first word.

We do not wait for the finished paragraph. By then the model has already chosen a path, and each next word makes that path harder to leave. So we look at the last input token, before any answer has been written. Everything gathered about the question sits in one state, and each possible next word is scored by how far that state already points toward it.

Most of those words start an answer. A few start a deferral, such as “I don’t know”. The contest is between the strongest of each kind, and the margin is how far the answer token sits ahead of the stop token.

\Delta(x) = z_{\text{answer}}(x) - z_{\text{stop}}(x)

Definition · Commit-abstain margin

The commit-abstain margin $\Delta(x)$ is how far the strongest way to start an answer sits ahead of the strongest way to start a stop, read at the last input token. Above zero, the model would rather begin an answer. Below zero, it would rather stop.

Above zero, the model would rather begin an answer. Below zero, it would rather stop. The lean is visible before a single word is generated. One number still hides who made it. Every attention head and every feed-forward block adds its own push, $v_k$ : positive toward an answer, negative toward a stop. Add those pushes, with $\kappa$ for the embedding and the bias, and the margin returns.

\Delta(x) = \kappa(x) + \sum_{k=1}^{K} v_k(x)

The sign of $\Delta(x)$ agrees with the model’s finished answer in 97.6% of cases. And it already separates cases where the model has enough evidence from cases where it does not, with a mean AUROC of 0.811. The warning is in there. On about 27% of questions without enough evidence, the model still prefers to answer, $\Delta(x) \ge 0$ . That is the gap the rest of the paper tries to open.

Margin distributions. Questions without enough evidence shift left, and a hatched mass remains at or above zero. — Questions without enough evidence shift left. The hatched region is still $\Delta(x) \ge 0$ .

Finding A

The margin $\Delta(x)$ reflects a reliable signal of whether the evidence is enough, yet unsupported commitment persists.

Then we ask which parts refuse to be switched off.

Most of the network can be switched without the decision moving. We want the parts that refuse. The weights stay frozen. Each attention head and each feed-forward block gets one learnable number, and only those numbers are trained.

Definition · Commit-Abstain Circuit

The Commit-Abstain Circuit (CAC) is the small set of attention heads and feed-forward blocks that causally carry the margin. Most of the rest can be switched without the decision moving.

That number has to act like a volume, somewhere between off and fully on. A sigmoid turns any real logit $g$ into a gate in that range, and the component’s output is scaled by it.

G = \sigma(g)

The gates need a reason to move. On a question with enough evidence, $y = 1$ , the margin should grow. On a question without it, $y = 0$ , the margin should shrink. The sign $(2y-1)$ is $+1$ in the first case and $-1$ in the second, so one loss pulls both ways. Minimising it keeps the gates that widen the right gap.

\mathcal{L}_{\mathrm{task}}(G) = -\frac{1}{B}\sum_{x}(2y-1)\,\Delta(x \mid G)

Left on its own, that loss leaves every helpful part switched on, and we still cannot see which parts the decision needs. A second term applies a pressure we can reverse, from the same starting point. $\lambda > 0$ pushes every gate open. $\lambda < 0$ pushes every gate closed. The clip keeps one huge logit from swallowing the penalty, so the pressure stays on all of them.

\mathcal{L}(G;\lambda) = \mathcal{L}_{\mathrm{task}}(G) - \lambda \sum_{g \in \mathcal{S}} \mathrm{clip}(g,-C,C)

A part that stays on when told to switch off, $G^- > \tau$ , is pushing toward an answer. A part that stays off when told to switch on, $G^+ < \tau$ , is pushing toward a stop. Everything that simply follows the pressure is left out. What remains is small. Across ten models, from 3B to 14B, the median circuit is 5.2% of the parts, and about a quarter of those are feed-forward blocks rather than attention heads. Every model we tested had one.

Causal gating scatter for Llama 3B. Commitment-promoting components resist removal. Abstention-promoting components resist retention. — Llama 3B. $G^- > \tau$ wants to answer. $G^+ < \tau$ wants to stop.

Circuit maps for ten models. Blue points are commitment-promoting. Red points are abstention-promoting. — The same kind of circuit in all ten models. Blue answers. Red stops.

Finding B

Every model we tested contains a sparse commit-abstain circuit.

The stop signal arrives after the answer has momentum.

Follow a question the model will rightly refuse, and one it should refuse but does not. For most of the depth they are the same story. Both accumulate a lean toward answering. A person does this too: the sentence starts, and only then does the doubt arrive.

The correction shows up after about $\ell/L = 0.80$ . When the model truly abstains, that late push is strong enough to cross back through zero. On the Earth question it is not. The answer already has momentum, $\Delta(x)$ stays positive, and the model commits. The figure below is the measured version of that story. Purple is a false commitment. Red is a true abstention. They travel together, then only one of them turns around.

Cumulative circuit contribution by layer depth. False commitment and true abstention rise together until 80 percent of depth, then only true abstention crosses below zero. — Purple is false commitment. Red is true abstention. They part after $\ell/L = 0.80$ .

Finding C

Commitment generally emerges in earlier layers than abstention.

The two pushes are not equal.

Finding the parts is not the same as showing they cause the decision. So we change them. Turning the answer-pushing parts down blurs whether the model has enough evidence: mean AUROC falls by 0.17. Turning off the same number of unrelated parts changes it by only about 0.069. The stopping parts are different. Turning them down barely moves the ranking. They nudge borderline cases. They are not what holds the distinction together.

Turning the parts up makes the imbalance obvious. A commitment direction can drive false abstention to zero: the model answers even when it does not have enough evidence. An abstention direction, scaled by $\alpha$ into the hundreds and then a thousand, still leaves false commitment well above zero.

At that scale the nudge becomes a test of the direction itself. The next layer normalises length. Make one part far larger than the residual $R$ it is added to, and the rest of the stream drops out of the normalised result. The layer then sees that part’s own direction $\hat{h}$ . A stop direction, seen almost alone, still cannot force a stop. A commitment direction, under the same wash-out, can force an answer.

\mathrm{RMSNorm}(R + \alpha h) \approx \hat{h} \quad \text{when} \quad \alpha \gg \frac{\|R\|}{\|h\|}

Ablation. Commitment-promoting components drag AUROC down. Abstention-promoting components barely move it. Grey lines are random. — Blue falls when commitment parts are removed. Red barely moves. Grey is random.

Amplification. False abstention falls toward zero when commitment components are scaled. False commitment stays above zero when abstention components are scaled. — Red falls toward zero as commitment parts are scaled by $\alpha$ . Blue does not.

Finding D

The parts that push toward an answer sustain the structure of the margin. The parts that push toward a stop correct it, and are often too weak to overturn the commitment already built up.

Seeing the pushes separately is enough to stop.

The last question in the paper is whether this is useful, not only descriptive. The margin adds every push into one number, and that sum can hide a reason to hesitate. A small classifier reads each push, $v_k(x)$ , on its own, and abstains when $P(\text{answerable})$ falls too low. The language model is not retrained. Accuracy moves from 69.2% to 81.4%. False abstentions fall from 32.0% to 12.7%, about 2.5 times fewer, without refusing when it did have enough evidence. A straight line through the same features gets most of that gain. On 24 of 24 held-out checks, two unseen benchmarks and two larger models, it beats the raw margin by 6.9 accuracy points on average. The information was in the circuit. The single number had been adding it away.

This is the decision to answer without enough information or evidence. It is not every way a model can be wrong.

Remark

The model already carries a signal that the evidence is not enough (Finding A), and it contains a sparse circuit (Finding B). It may still answer, because the stop arrives later (Finding C) and is often too weak to drive $\Delta(x)$ below zero (Finding D). The model commits not because a reason to stop is absent, but because that correction fails to overturn the commitment already accumulated.

Vy Nguyen, Ziqi Xu, Jeffrey Chan, Estrid He, Feng Xia, Renqiang Luo, Erik Cambria, Xiuzhen Zhang.

NeurIPS 2026. RMIT University, Jilin University, Nanyang Technological University.

Paper PDF Code

@inproceedings{nguyen2026commit,
  title={The Commit-Abstain Circuit: Why Language Models Hallucinate Instead of Abstaining},
  author={Nguyen, Vy and Xu, Ziqi and Chan, Jeffrey and He, Estrid and Xia, Feng and Luo, Renqiang and Cambria, Erik and Zhang, Xiuzhen},
  booktitle={Advances in Neural Information Processing Systems},
  year={2026},
  note={Accepted},
  url={https://arxiv.org/abs/2609.32964}
}