독서리서치레이더투자 프레임워크
로그인 / 회원가입
← 읽기로 돌아가기
장기 리딩중대2 분 소요DisconfirmAI 편집부

Machines That Learn Actions by Trial and Error Systematically Overstate Their Scores

Getting machines to learn continuous actions by trial and error—how much to press the gas, how much to rotate a joint—is called reinforcement learning.

검증됨 2026-10-05 00:00 GMT+8

Getting machines to learn continuous actions by trial and error—how much to press the gas, how much to rotate a joint—is called reinforcement learning. The machine has to maintain an estimate throughout: how many points it can get by following the current way of acting. The 2018 paper TD3 found that this estimate tends to be biased upward, and the machine chases the inflated score; the fix is to maintain two independent scorers, trust only the lower one, and slow down changes in the way of acting.

Today's demos of robots learning to walk and grasp things are often still backed by this trial-and-error learning, so the problem of inflated scores is still there. TD3's "take the lower of two scorers" was not bypassed by later new methods; instead, it became a standard feature of mainstream algorithms. Any system that chooses actions by estimating scores should ask one question: has the overestimation been prevented.

If the actions are discrete—choose left or right, rather than how many degrees—then don't use it to judge; its validation is entirely on simulated robot tasks, so don't directly extrapolate the results to real robots or other tasks. And suppressing the scores has a cost: genuinely good opportunities are also suppressed along with it, and overestimation is merely replaced by underestimation.

《Addressing Function Approximation Error in Actor-Critic Methods》(2018)|Next review 2027-09-20

출처

  • arxiv.org《Addressing Function Approximation Error in Actor-Critic Methods》(2018)
로그인 / 회원가입
독서리서치레이더투자 프레임워크
↗
리서치 계속하기 →