Cohere's CALIBER: One Confidence Score Hides Two Different Questions
AI company Cohere published CALIBER on October 20, 2026, arguing that reasoning models should estimate 'how likely am I to solve this?' before they begin and 'how likely is this answer to be correct?' after producing one. A single confidence number compresses these two questions into one, making calibration appear perfect while leaving users unable to identify which answers need scrutiny. CALIBER trains initial confidence on multi-attempt success rates and final confidence on individual answer correctness, reducing the 7B model's expected calibration error by 52.5% on a math benchmark and generalizing to three QA datasets without additional training.
Roll two fair six-sided dice. What is the probability that their sum is at least 10? A model might hesitate at first, then count the possibilities: three ways to roll 10, two ways to roll 11, one way to roll 12 — six outcomes out of thirty-six, giving 1/6. In this illustrative example, the model's confidence rises from 25% to 88%. Those percentages describe its confidence in getting the answer right, not the probability of the dice outcome. Working through the problem changed its assessment.
But confidence can move the other way too. Reasoning through a solution that initially looks straightforward may reveal a mistaken assumption or a contradiction. A useful confidence estimate should be able to reflect that change, even when the model still gives the wrong answer. However, standard reasoning models do not necessarily come with this ability: producing a detailed reasoning process does not guarantee that a model can reliably assess and communicate its uncertainty.
AI company Cohere published CALIBER on October 20, 2026, targeting exactly this problem. Its core thesis can be stated in one sentence: one confidence score hides two different questions.
Before reasoning, the model has only seen the question. It can recognize the topic and judge whether the problem looks manageable. Its confidence answers: 'How likely am I to solve this?' After reasoning, it has something more specific to assess: the solution it actually produced. Now the question is: 'How likely is this answer to be correct?'
These predictions are related, but they serve different purposes. Initial confidence tells a user whether the model expects to handle the question successfully; final confidence helps a user judge whether this particular answer needs closer scrutiny. A model might often solve this kind of problem and still make a mistake on this attempt. Or it might approach a difficult question cautiously, then find a solution it can check.
CALIBER's training mechanism directly embodies this separation. Initial confidence's training target comes from multiple attempts: let the model try the same question several times, and if seven out of ten succeed, 70% becomes the target. That captures how often the current model can solve the question, before we know which particular attempt it will produce. Final confidence gets a different target: whether that individual answer is correct. A successful and an unsuccessful attempt at the same question share an initial-confidence target but receive different final-confidence targets.
Why is reporting a single number not enough? Cohere illustrates with a thought experiment: imagine a model that answers 70% of questions correctly and reports 70% confidence every time. Its confidence matches its overall accuracy perfectly — but if you only have time to check a few answers, those identical scores give you no way to choose. Now imagine another model with the same 70% accuracy that assigns 90% confidence to half its answers and 50% to the other half. If those groups are correct 90% and 50% of the time respectively, this model is also calibrated. But its confidence is more useful: it helps you focus your checks on the answers more likely to contain mistakes.
Evaluation therefore requires looking at two properties together. Expected Calibration Error (ECE) measures whether confidence matches observed accuracy: among answers assigned roughly 80% confidence, roughly 80% should be correct. AUROC measures whether correct answers tend to receive higher confidence than incorrect ones. Lower ECE and higher AUROC are better.
In Cohere's self-reported results, the 7B model was trained on 30,000 math problems and evaluated on 1,000 held-out problems plus three question-answering benchmarks. On BigMathDigits, CALIBER's ECE falls from 0.059 for the strongest single-confidence baseline to 0.028 — a 52.5% reduction. AUROC reaches 0.806 and Brier score 0.130. More notable is the generalization: without further training or task-specific checkpoint selection, the 7B model reduces calibration error by 26.8% on GPQA, 32.2% on TriviaQA, and 36.2% on SimpleQA.
The 30B results show why the metrics matter together. On BigMathDigits, CALIBER's ECE of 0.022 is only slightly better than CoCA-lite's 0.023. But its AUROC is 0.797 compared with 0.508 for CoCA-lite — a large difference in how well confidence distinguishes correct from incorrect answers. Other baselines achieve slightly higher AUROC than CALIBER, so the benefit depends on which property you measure.
The difference between the two estimates carries information too. On BigMathDigits, the 7B model's mean confidence rises from 78.5% to 83.5% for correct answers and falls from 62.6% to 54.5% for incorrect answers. On other datasets, it sometimes lowers confidence for both groups, with a larger decrease for incorrect answers. A rise or fall is therefore context, not a verdict: two answers that finish at 70% confidence express the same estimated probability of correctness, even if one estimate rose and the other fell.
Cohere explicitly acknowledges the boundaries of this work: experiments measure confidence before and after a single reasoning process, and have not tested whether reliable confidence updates could help a model decide when to verify, retrieve more information, or abstain. CALIBER provides a starting point — confidence trained to reflect what the model can assess at each stage of reasoning.
All performance figures are Cohere's self-reported first-party benchmarks with no independent replication yet. The method's conceptual contribution — decomposing confidence into two distinct training targets — is logically independent of the specific numbers, but practical deployment value depends on whether those numbers hold under broader conditions.