Re:Learning ‘Montezuma’s Revenge’ from a single demonstration
by ilyasut on Wednesday July 04, 2018 @03:27PM
(#17459012)
There has been a fair bit of past work exploring the idea you described (examples: https://arxiv.org/pdf/1606.01868.pdf, https://arxiv.org/pdf/1703.01310.pdf, https://pathak22.github.io/noreward-rl/resources/icml17.pdf, https://openreview.net/forum?id=H1RPJf5Tz). Such methods can't solve games like Montezuma's revenge to a comparable level of performance yet, but I'm sure they'll eventually get there.
Re:Learning ‘Montezuma’s Revenge’ from a single demonstration
by Eridrus on Wednesday July 04, 2018 @05:34PM
(#17459552)
As the sibling comment says, there is work on this, but the key question becomes "how do you define novelty?" You could say new rooms are novel, but this is basically just the engineering approach of defining a new reward, and isn't really an interesting solution since it's not really applicable to other RL use cases, or even other games.
Re:Learning ‘Montezuma’s Revenge’ from a single demonstration
by dane-pgp on Wednesday July 04, 2018 @08:38PM
(#17460185)
In the worst case, the AI could store every single input it has received (for example, every frame it has seen) and calculate how similar each new input is to its past corpus.
Calculating similarity of images is quite a well-understood problem, but you're right that generalising the idea of similarity across all types of input data, in a way that is helpful for the AI to learn from, and efficient to calculate, may end up requiring a lot of coding that's specific to the individual use case.
Re:Learning ‘Montezuma’s Revenge’ from a single demonstration
by FartyMcFarter on Wednesday July 04, 2018 @06:48PM
(#17459839)
> and isn't really an interesting solution since it's not really applicable to other RL use cases, or even other games.
I think it applies to many games: you typically progress through different screens/levels/whatnot on your way to completing the game.
So they solved this by feeding the AI with a human demonstration, but have there been any attempts at giving the AI an explicit reward for maximising the "novelty" of the input state (i.e. the image on the screen)?
The game does not give the player points for reaching new rooms, but if the AI was rewarded for producing the "novel" state of a new room, then that would give it a drive to explore. Similarly, there would be an implicit penalty to the AI for repeatedly falling off a ledge or returning back to a room it had already visited (although some amount of back-tracking would no doubt be useful), whereas reaching a new part of the screen (by climbing a ladder, say) would be rewarded.
There are times where the AI would have to be patient and wait, but the window could be learned or set as a hyper-parameter. This might be enough to stop the unproductive behaviour of it jittering left and right continuously, since doing so does not produce a new state, relative to just standing still at least.