News for hackers. Stuff that matters.
Re:Specification gaming: the flip side of AI ingenuity by klmadfejno on Wednesday April 22, 2020 @01:37PM (#22948092)
Why can't you just train a model to predict the future and a model to play the game; then have the game playing model take the course that leads to the least certain outcome until the model can comfortably identify game states, at which point you just say the reward function is proximity to that game state? Seems to me the problem with trying to define a reward function up front is that the AI has no clue how to describe game state so even if you do a good job you're just side stepping the actual problem.
Re:Specification gaming: the flip side of AI ingenuity by LolWolf on Wednesday April 22, 2020 @02:29PM (#22948658)
> leads to the least certain outcome

What's the loss function for this? How do you cross-validate this loss? The only generally applicable methods I can think of (essentially some Bayesian-type updating, which then uses Monte-Carlo-type methods for evaluating this loss and "uncertainty") is just so much more expensive than anything we do at the moment that, at the current computing level, we wouldn't even be able to do it in most cases.

The problem is, an AI model will tell you an "uncertainty" but these "uncertainties" will often not reflect the true underlying probability that it is wrong. This is, of course, all a bit silly since it's hard knowing what you don't know, and a proper update to this number is also not well-defined unless you already have a bunch of samples about potential future outcomes (via, like I mentioned before, a Monte-Carlo type approach. This would multiply the cost of something as simple as an evaluation by many times.)

Re:Specification gaming: the flip side of AI ingenuity by klmadfejno on Wednesday April 22, 2020 @02:57PM (#22948955)
In my mental model you don't need a loss function for this. It's just, given these potential short term goals predict the future state of the game 1 minute from now. Then choose the short term goal that maximizes uncertainty for the future. The idea being that you have a game playing bot that learns to do "things" in the game world, and a game state encoding bot that learns to describe the game world. I'm not suggesting you train the bot TO do the least certain idea, I'm saying you train the bot BY doing the least certain idea; and update your understanding of the world to be more certain about that choice in the future.

The measure of uncertainty could be as simple as a multi armed bandit where the reward is % of accurate guesses following that path.

The pseudo adversarial aspect of this being that both models will learn to agree on how choices translate to gameplay changes. The bot will not learn to beat the game necessarily, but a Style GANN kind of approach would probably be just fine to saying, "Please get to a state that looks like this" with a few manual samples. Perhaps all of this to say, it's better to learn how to play the game, and specify that, in this particular run, you'd like the bot to try and reach the game end credits.

Re:Specification gaming: the flip side of AI ingenuity by johnmoberg on Wednesday April 22, 2020 @03:25PM (#22949227)
Your last paragraph reminds me of Hindsight Experience Replay[1] where they use goal-conditioned policies that learn how to reach essentially any state. The idea is that, even if you don't reach your true goal, you've learned how to reach some other goal, so you can successively learn to navigate the environment until the end goal is easy. It's a really nice paper, highly recommended!

[1] https://arxiv.org/abs/1707.01495

Re:Specification gaming: the flip side of AI ingenuity by johnmoberg on Wednesday April 22, 2020 @03:19PM (#22949163)
You might like [1], where they use an ensemble of models and essentially optimize for novelty as measured by the entropy of model predictions across the ensemble.

[1] Model-Based Active Exploration, Shyam et al. 2019, https://arxiv.org/abs/1810.12162