Confidence Comes From Experience: What XConf Changes About How We Measure LLM Confidence

Author(s): luisacsfreitas

Originally published on Towards AI.

Anyone who has put an LLM in charge of a real decision knows the question that comes right after the demo: when do we trust it? Routing an email to the right team, approving a patch, answering a customer without review. In all of these we need a number that says “this one can go, this one goes to a person”.

The trouble is that the number usually comes from the model itself, and the model is not a good judge of itself.

On 15 September 2026, a team from the University of Cambridge and Google DeepMind published “Confidence Comes from Experience: Experiential Confidence Estimation from Reasoning to Agents” (Zhang, Zhu, Li, Chen, Kumaran and Collier). The method, called XConf, is simple to state and changes the starting point: confidence is not read from the current answer. It is read from the history of past answers.

In this post I explain the idea, what the results actually show, how it compares with the rest of the confidence-estimation toolbox, and what it takes to use it in a real system, including the risks the paper does not cover.

TL;DR

The usual methods (verbalized confidence, token probabilities, self-consistency) only look at the current answer. When a model is wrong the same way every time, all of them say “confident”.
XConf keeps a bank of past, graded episodes and asks: on similar tasks, when the model was as sure as it is now, how often was it right?
With a single generation, it matches or beats ten-sample self-consistency in 23 of 24 comparisons, with much lower calibration error.
It needs no logit access, trains nothing, and works the same for multiple choice, code and agents.
The one condition everything rests on: the right/wrong labels must come from outside the model. With labels produced by the model itself, the method gets worse.

1. The problem: confidence only reads the moment

There are three families of methods for putting a confidence number on a black-box LLM:

Verbalized confidence. Ask the model: “from 0 to 1, how sure are you?”. Cheap and works on any task, but models are systematically overconfident and sensitive to how the question is phrased.
Token probabilities (logprobs, P(True)). The probability the model assigned to its own answer. It requires logit access, which most frontier APIs do not give, and it measures uncertainty about the next token rather than about whether the claim is true.
Self-consistency. Ask several times with temperature and see whether the answer changes. It was the strongest black-box method available, but it costs N generations per request and does not map cleanly onto code or agent trajectories.

All three have one thing in common: they only look at what the model is doing right now. And they share a blind spot. If the model has a stable misconception (it reads “stop paying” and always thinks payments, when in your company that is a cancellation), it gives the same answer every time, with the same certainty and high probabilities. All three methods report “confident” on an error.

Legenda: Figure 1. Left: the usual methods only read the current answer. Right: XConf reads the history of similar episodes.

2. The idea: confidence as an observed frequency

The paper draws on decades of research on human metacognition. We do not judge our confidence only by re-inspecting the reasoning we just did. We also remember how similar situations turned out.

A student who has solved a hundred determinants trusts the result without re-checking. The same student, facing a hard inequality, writes the answer already expecting it to be wrong, because they remember how often such proofs collapsed on the last line.

XConf formalises this. Instead of asking the model what it feels, it asks the record what happened.

3. How it works

Figure 2. Two readings of the same record: Recall (statistical) and Reflect (verbal).

The experience bank. Every time the model solves a task, an episode is stored with five fields:

the task;
a short reflection by the model on its own solution, written before the outcome is known;
the confidence the model stated;
the graded outcome (right or wrong), once it arrives;
a lesson, written by the model after it learns the outcome.

The lesson is quarantined: it is never shown to the model while it assesses a solution that has not been graded yet. The authors are explicit about why: knowing the outcome biases self-judgement in ways that instructions alone do not fix.

Recall: a confidence-conditioned hit rate. For a new task, XConf retrieves the most similar episodes from the bank. The important detail is that the search key has two parts: the task embedding and the stated confidence. It does not look for “similar tasks”, it looks for “similar tasks where the model felt equally sure”. It takes the 50 nearest neighbours and computes the fraction in which the model was right.

The paper shows this conditioning is not decoration: removing stated confidence from the key costs 0.08 AUROC on reasoning and 0.12 on agents. There is one more subtle detail: similarity is not raw embedding cosine. It is measured in a space rescaled using the bank’s own graded outcomes, so that “similar” means “fails for the same reasons” rather than just “is about the same topic”.

Reflect: the model reads its own track record. The retrieved neighbours are shown to the model as short cards (task, stated confidence, outcome, lesson). The model first has to name the recurring failure mode it sees, and only then restates a confidence. It is a short call that does not re-solve the task.

Combine. The final confidence is the plain average of the two readings: ½ × (Recall + Reflect).

Two properties make this practical. When the bank is sparse around a task, the neighbours are only weakly similar and Recall relaxes towards the model’s base success rate at that confidence level: it degrades gracefully instead of failing. And nothing depends on the output format. A multiple-choice letter, a 200-line program and a 30-step agent rollout are stored in exactly the same way.

4. An easy example

Picture a system that routes customer emails at an insurance company to the right team. This one arrives:

“I sold my car. How do I stop paying for the insurance?”

The LLM reads “stop paying” and answers Payments, with high confidence. Any method that only looks at this answer agrees: the sentence is clear, the model is sure, and if you ask ten times you get the same answer ten times.

XConf goes to the record, finds similar emails where the model also said “Payments, high confidence”, and checks where they ended up after a person handled them.

Figure 3. The same reasoning applied to one email. The model is sure, but its track record says that, for this kind of request, it is usually wrong.

Out of 10 similar cases, only 4 actually stayed in Payments. The rest ended up in Cancellations, because in practice people who write “stop paying” want to cancel. Recall = 0.40. Reflect, after seeing the cards, names the confusion and lowers it further. The final probability lands well below the threshold, and the email goes to human review instead of going straight to the wrong team.

For contrast, “Can I pay by bank transfer?” also comes out as “Payments, high”, but there the record says 48 out of 50 were right, and the email goes through untouched.

No new rule had to be written. The system learned to distrust the model exactly where it has a conceptual error, just by recording where the emails ended up.

(The numbers in this example are illustrative. The paper uses 50 neighbours; the figure shows 10 to fit.)

5. What the results show

The evaluation covers nine benchmarks (multiple choice, hard reasoning, maths olympiads, code, multimodal questions and three agent environments) and four models from three families: Gemini 2.5 Flash, Gemini 3.5 Flash, Claude Sonnet 4.6 and Qwen3.5–397B. The protocol is careful: five-fold rotation, so the bank behind each estimate never contains the task itself. The authors also replay the protocol with the bank growing in strict chronological order, as it would in production.

Figure 4. AUROC (ability to separate right from wrong answers) for one of the models. One XConf generation against ten self-consistency samples.

The numbers I find most relevant for people building systems:

Against ten-sample self-consistency: 21 wins, 2 ties and 1 loss, at a tenth of the generation cost. Calibration error is lower across the board (on MMLU-Pro, three to eight times lower).
The gap is largest on agents, because that is where failures are silent: a failed trajectory looks clean. On AppWorld, verbalized confidence averages 0.72 AUROC and retrieved experience reaches 0.86. On SWE-bench Verified, it beats a purpose-trained verifier.
For deciding what to automate: abstaining on the 10% least-confident cases raises the accuracy of what is delivered by 4.8 points on average, and by up to 8.7 points on AppWorld. The most-confident decile is right 98% of the time (versus 90% for verbalized confidence and 87% for self-consistency), and the least-confident fifth holds half of all errors.
It is not just “this question is hard”. With question difficulty controlled for, XConf still separates right from wrong at 0.79 AUROC.
It improves with volume. As the bank grows, calibration keeps improving. On AppWorld with Claude, AUROC goes from 0.63 to 0.81. On reasoning, a few hundred episodes already match self-consistency wherever verbalized confidence is weak.

6. Where this fits: is it really new?

Measuring a historical hit rate is not a new idea. Classical calibration (Platt scaling, histogram binning, isotonic regression) does exactly that at the level of a score: “when the model says 0.9, how often is it right?”. Neighbour-based methods such as the Trust Score (Jiang et al., 2018) and local variants of conformal prediction already asked “on similar examples, how reliable is the classifier?”.

Download the Medium app

What XConf brings is the combination, applied to today’s LLMs:

it conditions retrieval on the confidence the model states, which is the only internal signal available through a closed API;
it works without logits, without training and on any output format;
it adds a verbal reading (Reflect) that uses the model’s own ability to spot patterns in its history;
and it shows, with measurements, that this beats the sampling-based methods that were the reference point.

Figure 5. How confidence-estimation families compare. The first four columns follow Table 1 of the paper; the last two are my own assessment.

An honest reading of the table: post-hoc calibration and trained verifiers also use graded outcomes. The difference is that they need refitting or retraining to benefit from new data, whereas a new episode in the XConf bank counts from the moment it is stored.

7. Architecture in production

The paper evaluates the method on its own. In a real system I would not run it alone: I would add Recall as one more signal next to the ones you already have, and keep Reflect for the borderline cases.

Figure 6. A production architecture: Recall runs on every request, Reflect only in the grey zone, and the bank is fed by real outcomes.

Online (per request). The model answers with a single generation that includes its stated confidence (a discrete scale such as “low / medium / high” works better than 0–100). Recall is an in-memory lookup that does not call the model. A simple calibrator (logistic regression is enough) combines Recall with any other signals, and the result is compared with a threshold.

The threshold is not picked by eye. Decide the error rate the business accepts on automated cases (say, at most 3%) and derive the threshold from a calibration set. That is the logic of selective prediction and conformal risk control.

Offline (when the outcome arrives). Every real outcome becomes an episode: a human fixed it, a test passed, a ticket was rerouted, an independent judge graded it. A periodic job writes the lessons, removes duplicates and refits the calibrator.

Three variants, from simplest to most complete:

Recall only. Zero extra calls. The best option with small models: the paper shows the statistical reading barely degrades with model size, while Reflect loses quality below roughly 27B parameters.
Recall everywhere, Reflect in the grey zone. One extra call only for cases near the threshold. This is the trade-off I would pick.
XConf as in the paper. Average of Recall and Reflect on every request. Simplest to set up, but you pay the extra call every time.

Recall itself is a few lines:

import numpy as np

def recall(q_emb, q_conf, bank, k=50, min_neighbours=20):
"""Hit rate on similar episodes with the same stated confidence."""
m = bank.conf == q_conf # same confidence level
if m.sum() < min_neighbours:
return bank.correct[m].mean() if m.any() else None # fallback: base rate
sims = bank.emb[m] @ q_emb # normalised embeddings
top = np.argsort(-sims)[:k]
return bank.correct[m][top].mean()

(Simplified, plain cosine. The authors’ code is at github.com/caiqizh/xconf.)

8. Use cases

It pays off most where the real outcome arrives anyway and failures are silent:

Triage and routing of emails and tickets. The outcome is free: if someone rerouted the case, the model was wrong.
Agents that must choose between shipping, escalating or retrying. This is where the paper shows the biggest gains.
Coding agents. Tests, reviews and reverts provide the outcome.
Document extraction (invoices, forms), where a later validation confirms or corrects the fields.
Answers over internal knowledge bases, when there is explicit user feedback or sampled review.

Where it is not the best choice:

Short factual questions, where voting still wins. The authors acknowledge that self-consistency keeps the edge on this kind of task.
Tasks with no verifiable outcome, such as creative writing or opinion.
Brand-new domains with no history. There XConf falls back to the base rate, and you need another signal until the bank fills up.

9. Risks and caveats

Most of these are not in the paper. They come from thinking about the method inside a real system.

Labels must be independent of the model. This is the most important point, and the paper measures it. A bank labelled by an LLM judge that agrees 91% with the gold labels keeps most of the value. A bank labelled by the model itself ends up worse than having no labels at all, because self-judgements are wrong precisely on the confidently-wrong cases the bank is meant to catch.

Cold start. No history means no Recall. Before asking anyone to label, look for operational signals you already have: reroutes between teams, reopened tickets, failed tests, manual corrections. Run the model over old closed cases and compare with where they ended up. That gives you a starting bank with zero annotation cost.

Selection bias after you automate. Once confident cases go through on their own, only doubtful ones reach people, and the bank fills up with hard cases. Keep a random sample of automated cases under review, and weight it accordingly.

Changing models. The bank describes the errors of one model. The paper measures that another model’s record loses 0.03–0.06 AUROC on the same tasks. Store the model version with each episode and treat a version change like a deploy.

Leakage. Emails from the same thread, or near-duplicates, find themselves in the bank and inflate the results. Deduplicate, and split by thread or by customer when you evaluate.

Personal data. The bank stores real tasks, so it stores real data. Mask what you do not need, set a retention policy, and remember that an embedding is personal data too.

The paper is new. It is a week old, the numbers are the authors’ own, and there is no independent replication yet. The evaluation uses benchmarks, not a production system. Validate it on your own replay before trusting it.

10. How to start

Make sure that, for every decision the model makes, you know the real outcome. Without this there is nothing to do.
Ask the model for a confidence on a discrete scale, in the same call as the answer.
Store each decision with the task embedding, the answer, the confidence and the model version.
When the outcome arrives, mark it right or wrong using a source independent of the model.
With a few hundred episodes, compute Recall and measure its AUROC against what you use today.
Add it to your signals in a calibrator and set the threshold from the acceptable error rate.
Only then, if the grey zone is large, try Reflect on those cases.

Conclusion

What convinces me most about XConf is not the win count against self-consistency. It is the change of question. For years we tried to extract confidence from inside the model, through probabilities, samples and introspection. XConf accepts that the model does not know where it goes wrong, and gets that information from something that does: the record of what happened.

For anyone running LLM systems, the practical consequence is good news. Many of us already have that record, scattered across logs, reroutes and corrections. What is missing is storing it as experience and letting it decide when to trust the model.

References

Zhang, C., Zhu, X., Li, C., Chen, Y., Kumaran, D. and Collier, N. (2026). Confidence Comes from Experience: Experiential Confidence Estimation from Reasoning to Agents. arXiv:2609.17708. · Code:
r/cap summary of the paper (15 Sep 2026):
Frohn, S. (2026). The Impact of LLM Self-Consistency and Reasoning Effort on Automated Scoring Accuracy and Cost. arXiv:2604.26954.
Zhang, C., Yang, R. et al. (2026). Confidence Estimation for LLMs in Multi-turn Interactions. Findings of ACL 2026.

Figures by the author, built from the results published in the paper. The email examples are illustrative.

Published via Towards AI

Source link