>This is a failure of the "game" rules, provided the intended outcome is something other than this (as it surely was/is). Making a game with winners and losers and any kind of objective scoring system out of something resembling a real activity is really hard to do without distorting that activity until it no longer resembles what you wanted it to.

Right. In practice, if you want to engineer the rules to be good, (I think) you need both the rigid, pre-defined, unforgiving logical rules, and some inarticulable "I know it when I see it" rules.

I wonder, are there any theorems on the bounds of how far an optimizer can deviate from another's agent's desired outcome if the agent is limited to specifying the rules in n bits and can specify p arbitrary vetoes?

(Made another reply, but kept this separate for its unrelated point.)