News for hackers. Stuff that matters.

mjaskowski

Re:Deep Q-Learning: Space Invaders on Tuesday March 15, 2016 @04:36PM (#11292779)
Actually I did both frame skipping and amxing out two frames as reported in the Natures letter

https://storage.googleapis.com/deepmind-data/assets/papers/D...

Re:Deep Q-Learning: Space Invaders on Tuesday March 15, 2016 @04:34PM (#11292763)
Absolutely. Q-learning has this capabilities and a shallow neural network was used back in 1992 to play backgammon, which has a lot of stochasticity. See https://en.wikipedia.org/wiki/TD-Gammon
Re:Deep Q-Learning: Space Invaders on Monday March 14, 2016 @04:27PM (#11285335)
Yes, we try to approximate Q function with neural network. Which is basically an enhanced version of gradient-descent Sarsa.

The main trick to notice is that you can't provide consecutive frames as mini-batches as these would be highly correlated and would derail stochastic gradient descent.

So we keep many frames (and all other necessary information) in memory and draw these experiences uniformly to form a minibatch that becomes input to the neural network

Re:Deep Q-Learning: Space Invaders on Monday March 14, 2016 @01:45PM (#11284230)
Let me fix that. There are actually two papers: https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf

and a more recent and more detailed: https://storage.googleapis.com/deepmind-data/assets/papers/D...

Re:Deep Q-Learning: Space Invaders on Monday March 14, 2016 @01:34PM (#11284140)
Note that Neural Network is just a very complex function.

You usually think of Q as a function (S, A) -> (Expected accumulated future reward)

which is equivalent to S -> A -> (Expected accumulated future reward)

the Neural Network is S -> (A -> (Expected accumulated future reward)) or if you whish the output layer of neural network consists of |A| neurons. Each indicates the (Expected accumulated future reward) given current experience.

Re:Deep Q-Learning: Space Invaders on Monday March 14, 2016 @01:11PM (#11283991)
I noticed that the video does not work in Safari.

Interesting. I am not sure how this deepmind game was played. Note that a typical game with epsilon = 0.1 achieves results around 550 points. In the nature paper they were running the game with epsilon = .05

If you spot any other differences I'd be happy to learn about them.