It is very difficult to safely define a loss function that adequately represents not what you believe now you want, but rather the outcomes that you would be happy with in hindsight once they occur. Especially if the rules of the game are complex enough and allow for "creativity".

https://www.lesswrong.com/posts/4ARaTpNX62uaL86j6/the-hidden...