Cohere's CALIBER: One Confidence Score Hides Two Different Questions
AI company Cohere published CALIBER on October 20, 2026, arguing that reasoning models should estimate 'how likely am I to solve this?' before they begin and 'how likely is this answer to be correct?' after producing one. A single confidence number compresses these two questions into one, making calibration appear perfect while leaving users unable to identify which answers need scrutiny. CALIBER trains initial confidence on multi-attempt success rates and final confidence on individual answer correctness, reducing the 7B model's expected calibration error by 52.5% on a math benchmark and generalizing to three QA datasets without additional training.
Roll two fair six-sided dice. What is the probability that their sum is at least 10? A model might hesitate at first, then count the possibilities: three ways to roll 10, two ways to roll 11, one way to roll 12 — six outcomes out of thirty-six, giving 1/6. In this illustrative example, the model's confidence rises from 25% to 88%. Those percentages describe its confidence in getting the answer right, not the probability of the dice outcome. Working through the problem changed its assessment.
But confidence can move the other way too. Reasoning through a solution that initially looks straightforward may reveal a mistaken assumption or a contradiction. A useful confidence estimate should be able to reflect that change, even when the model still gives the wrong answer. However, standard reasoning models do not necessarily come with this ability: producing a detailed reasoning process does not guarantee that a model can reliably assess and communicate its uncertainty.
AI company Cohere published CALIBER on October 20, 2026, targeting exactly this problem. Its core thesis can be stated in one sentence: one confidence score hides two different questions.
Before reasoning, the model has only seen the question. It can recognize the topic and judge whether the problem looks manageable. Its confidence answers: 'How likely am I to solve this?' After reasoning, it has something more specific to assess: the solution it actually produced. Now the question is: 'How likely is this answer to be correct?'