독서리서치레이더투자 프레임워크
로그인 / 회원가입
← 읽기로 돌아가기
장기 리딩중대1 분 소요DisconfirmAI 편집부

When a model gives reasons, first check whether it's computing the right ledger

Some prediction systems give a reason for each decision: this loan was rejected because income was the factor with the largest share.

검증됨 2026-10-02 21:03 GMT+8

Some prediction systems give a reason for each decision: this loan was rejected because income was the factor with the largest share. SHAP is an algorithm that sets the rules for this kind of explanation; it breaks down and computes each factor's contribution to the outcome.

The explanation bar charts behind today's risk-control and medical models are mostly powered by this algorithm. The 2017 paper proved that if you want reasons to satisfy a few common-sense properties — each factor states its own amount clearly, unclaimed portions are accounted for, and the reasons don't change depending on how factors are grouped — only one algorithm can do it. Later explanation tools are basically patches on top of it.

If the model is complex and factors interact with each other, exact computation is impossible, and approximations rely on extra assumptions — don't use it to judge contributions when interactions are strong. Small-scale user experiments are also not enough to prove that people actually trust its explanations more.

《A Unified Approach to Interpreting Model Predictions》(2017) | Next review 2027-09-20

출처

  • arxiv.org《A Unified Approach to Interpreting Model Predictions》(2017)↗
리서치 계속하기 →
로그인 / 회원가입
독서리서치레이더투자 프레임워크