News for hackers. Stuff that matters.
Re:Deep Reinforcement Learning by seanwilson on Saturday June 18, 2016 @12:49AM (#11927165)
When it's playing a game (e.g. breakout) and it's being fed the pixels on the screen, how is the AI being told what the score/progress is? Does it have access to some numeric metric that is chosen by the researchers for each game?
Re:Deep Reinforcement Learning by sanxiyn on Saturday June 18, 2016 @01:14AM (#11927206)
Yes.

For example, Breakout saves score in address 76 and 77. Arcade Learning Environment has code to read the score, one per game. Code for Breakout is here: https://github.com/mgbellemare/Arcade-Learning-Environment/b...

Re:Deep Reinforcement Learning by seanwilson on Saturday June 18, 2016 @10:16AM (#11928699)
Thanks for that! So I was looking at the one for Montezuma's Revenge: https://github.com/mgbellemare/Arcade-Learning-Environment/b...

The reward seems to be only the game score which I believe is meant to be problematic for this game because your score doesn't go up very often (so you have to perform a lot of actions to get any feedback)? The lives are recorded but aren't part of the "get reward" method...are the lives factored into decision making somewhere else? Seems like knowing you just lost a life would really help decision making for such a game.

Re:Deep Reinforcement Learning by Phemist on Saturday June 18, 2016 @10:57AM (#11928878)
I think novel game situations were counted into the reward function as well
Re:Deep Reinforcement Learning by gwern on Saturday June 18, 2016 @01:10PM (#11929486)
Only in the specialty novelty-oriented DQN agents; the Montezuma's Revenge reward itself remains the same. The problem is defining 'novel' when every screen's pixels may be different (for example, imagine any game which has a timer ticking up).
Re:Deep Reinforcement Learning by Phemist on Monday June 20, 2016 @05:23AM (#11936990)
Not so much, a timer ticking up is only novel the first time round, and is unrelated to actions taken by the agent. Over multiple plays the agent will learn to ignore it.

EDIT: It could be that the agent will just stand there the first few plays around, enjoying the novelty reward gained from simply watching the timer tick up. Haha

Re:Deep Reinforcement Learning by gwern on Wednesday June 22, 2016 @05:21PM (#11957128)
The point is that every time the timer ticks, if you had defined 'novelty' as the bitstring representing the screen, you get a 'new' state. This multiplies against any blinking animations, any moving enemies, any of the agent's moves, any visible scores, etc. You get thousands or millions of unique framebuffer states before the agent has so much as left the first room in _Montezuma's Revenge_. And DQN already is RAM intensive for the experience replay buffer.
Re:Deep Reinforcement Learning by Phemist on Wednesday June 22, 2016 @06:32PM (#11957565)
Thanks for the extra explanation. It seems I assumed too much about these Deep Q networks, due to some prior knowledge of the neuroscience related to RL. Although I do remember having seen a video about Montezuma's Revenge a week ago or so, where they talked about this exact problem.

Anyway, it would seem to me that novelty functions that would allow the agent to ignore periodic changes in state such as timers going up, can be quite simple. A function that estimates novelty of individual bit values in the bitstring of the gamestate and then aggregates it, could quite easily account for timers, or generally elements changing periodically regardless of agent actions. A baseline novelty reward would seem relatively easy to predict by the agent and thus result in low prediction errors and low reinforcement of actions by agents. This function would have a linear space and time complexity to the length of the gamestate, and fairly naive & simple to use, but would get the job done I think?

P.S Just wanted to thank you for the work you've put into your website, it's very informative and always a great starting point to dive deeper into topics you cover!

Re:Deep Reinforcement Learning by gwern on Thursday June 23, 2016 @10:30AM (#11961118)
You have to come up with something or else the agent will never be able to explore worth a damn in complex domains. Imagine trying to learn to write Haskell programs by typing random gibberish...

'gamestate' is illegal. It's pointless to suppose an agent which has access to the true groundtruth RAM of the Atari games, because that generalizes to vanishingly few other domains. The goal is to create a general agent which can be used elsewhere, such as in recommender systems. (And if you did have access to the raw RAM, that would reduce the problem from an extremely challenging POMDP or harder, to a fully-observed deterministic MDP, because you could then construct a game-tree of each individual RAM state and the possible actions taken in it; in which case, you would use a much faster and more powerful MDP solver like MCTS rather than DQN.)

One can come up with hand-crafted heuristics which might improve over the naive bitstring equality approach, but your suggestion still doesn't do the trick, assuming you could figure out how to meaningfully define 'periodic changes' and teach the NN to ignore them. Imagine a game in which the overall screen lighting varies (perhaps it's set at night or during rain, or perhaps each level has different color themes). As all the bits keep flipping with changes in lighting/intensity, you'd be in about the same place.

Re:Deep Reinforcement Learning by msohcw on Saturday June 18, 2016 @01:15AM (#11927213)
Yes, it's fed the score as the reward value used. If I'm not wrong, they didn't normalise it across games for the initial paper but normalised it to some range for some of the following research experiments.
Re:Deep Reinforcement Learning by cosmoharrigan on Saturday June 18, 2016 @01:22AM (#11927225)
The original papers [1][2] for the deep Q-network used reward clipping: "As the scale of scores varies greatly from game to game, we clipped all positive rewards at 1 and all negative rewards at -1, leaving 0 rewards unchanged".

[1] https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf

[2] http://home.uchicago.edu/~arij/journalclub/papers/2015_Mnih_...

Re:Deep Reinforcement Learning by Dolores12 on Saturday June 18, 2016 @01:59AM (#11927293)
They can measure time you played. More time in game mean higher score.