Very Cool.

I did a similar project using a Neural Network instead of a look up table, similar to how deepmind did their Atari game system. The neural network adds some overhead to the training time because of backprop, so for small state spaces which the table can handle easily it is a bit overkill. But the neural network can scale up to much more complex state spaces that would be impractical to use a table for.

Do you have a library for your project? It sounds really interesting. I think this space is interesting enough that would benefit from a GutHub project to collaborate.

It looks based on this repo: https://github.com/nrox/q-learning.js and this example: http://nrox.github.io/q-learning.js/test2.html

that makes me proud :) do you have suggestions to improve this library ?

Food / Poison break even was at 88. A chart would be interesting.

However - since R2D2 seems like a slow learner it would be nice to speed up the time a little bit.

Warning: I only glanced through the code real quickly, so I haven't checked to see if this has unintended side effects to the game logic, or to make sure it's a complete solution.

It looks like there's a variable that controls how quickly the board itself is updated:

  var boardGameUpdateInterval = 10;

And a function deciding how quickly the visual representation gets updated:

  var render = function () {
      setTimeout( function() {          
          requestAnimationFrame(render);
      }, 1000 / 15);

      if (gameTicks % boardGameUpdateInterval == 0) {
          updateBoard();
      }
  
      animateBoard();
  
      gameTicks++;
      renderer.render(scene, camera);
  };
  

They're global variables so it's pretty easy to just drop into the developer console and overwrite them.

I just changed the denominator of the setTimeout to 30 instead of 15, and the boardGameUpdateInterval to 5 instead of 10. Of course, you don't have to change the render speed to get the logic speedup, but I figured I'd do both.

This appears to have sped the simulation speed up by a factor of two.

ymmv

I've gone as far as ~1000 food ~100 poisson blocks to make sure it's on the right track. Speeding this up is a good idea, I'll make some time to add this and post an update. You can ping me on twitter @javierluraschi.

After ~30mins I got ratio of 0.77 (375 greens, 111 blacks), it got pretty good at avoiding blacks after reaching 0.7.

I am currently at 175 vs 75

Looks very nice, I have one question:

How does R2D2 "see" the blocks? Is it by pixels or is the game hard-coded in this sense?

Currently, it sees a board of 3x3 states, as in:

_ _ G

G B B

_ _ G

A next step would be to use real computer vision over a 3D camera in the environment aligned to R2D@ and a deep neural network to interpret pixels instead of states. DeepMind has a paper describing this approach: http://arxiv.org/pdf/1312.5602.pdf

It seems like board is kept as a grid and merely represented with three.js; look into the page's source and you'll see a GameBoard object as well as its logic.