When a model gives reasons, first check whether it's computing the right ledger

Some prediction systems give a reason for each decision: this loan was rejected because income was the factor with the largest share. SHAP is an algorithm that sets the rules for this kind of explanation; it breaks down and computes each factor's contribution to the outcome.

The explanation bar charts behind today's risk-control and medical models are mostly powered by this algorithm. The 2017 paper proved that if you want reasons to satisfy a few common-sense properties — each factor states its own amount clearly, unclaimed portions are accounted for, and the reasons don't change depending on how factors are grouped — only one algorithm can do it. Later explanation tools are basically patches on top of it.

If the model is complex and factors interact with each other, exact computation is impossible, and approximations rely on extra assumptions — don't use it to judge contributions when interactions are strong. Small-scale user experiments are also not enough to prove that people actually trust its explanations more.

《A Unified Approach to Interpreting Model Predictions》(2017) | Next review 2027-09-20