To ask how confident a model is, have it answer several times and see how scattered the answers are

There is an old trick when training neural networks that make judgments automatically: at each step, randomly and temporarily turn off some units, known in jargon as Dropout. A 2015 paper explained mathematically that if you keep this randomness at test time and compute the same question several times, the disagreement among the answers is the model's uncertainty about itself.

Today models make decisions for people everywhere, and when they are wrong they still answer with certainty, so how much uncertainty they themselves have has become an unavoidable question. Later methods mostly require training several sets of models, while this one needs no retraining and only a few extra computations, and it is still the default starting point today. When vendors claim a model knows what it does not know, first ask whether it is done this way.

If this randomness was not turned on during network training, do not use it to judge confidence — there is no randomness to keep, and the method cannot be used. It has only been validated on small tasks such as predicting numerical values and handwritten digits. When the several answers scatter into several clusters, the confidence it gives is only a rough approximation, so do not treat it as a precise probability of error.

《Dropout as a Bayesian Approximation》(2015) | Next review 2027-09-20