In particular, the initial input to the neural networks is a 19×19×48 grid, and the layers of this grid include information like:
- How many turns since a move was played
- Number of liberties (empty adjacent points)
- How many opponent stones would be captured
- How many of own stones would be captured
- Number of liberties after this move is played
- Whether a move at this point is a successful ladder capture
- Whether a move at this point is a successful ladder escape
- Whether a move is legal and does not fill its own eyes
Again, before the neural nets even get involved. Some of these layers are repeated 8 times for symmetry. I would say for some of these, AlphaGo got some domain-specific help in a non-general way.
It is of course still groundbreaking academically. The architecture is a state-of-the-art deep learning setup and we learned a ton about how Go and games in general work. The interaction between supervised and reinforcement learning was interesting, especially how the latter behaved worse in practice in selecting most likely moves.
disclaim: Googler, not in anything AI.
The most theoretically interesting thing to me is the use of stochastic sampling to reduce the search space. Is there any discussion of how well pure Monte Carlo tree search performs here compared to the system incorporating extensive domain knowledge?
- I don't see what the problem is with that. Nearly all tasks humans would currently gain in getting automated have plenty of human experience before it. This is why training in any field even exists. Sure it would be cool to send a drone in unexplored territory with no knowledge and come back later to a self-built city, but I don't see how putting that milestone a bit later is in any way a problem.
- The hardware requirements for any new interesting breakthrough to do something important have always been immense. Just as for Deep Blue, or to all the graphite experiments where we can only produce just milligrams of it at a time, or to solar panel, each new discovery innovates in a specific direction. It is just a matter of time before the new method is substantially improved and will be able to run on single machines. To expect otherwise is foolish, to complain about it is pointless.
EDIT:
This is not to contradict the general point. That we don't need new breakthroughs is utterly false in so many ways it's not even funny. However this certainly is an AI breakthough, one that many AI researchers (myself included) thought it would take AT LEAST 10 years to come to pass. Breaking 10 years in 1 seems like a breakthrough to me.
+ The issue with MCTS was not that it couldn't be extended to non-determinism (you're correct re: POMCP) but that it requires a simulator which produces at least a reasonably accurate model of the world. This simulator is almost always hand engineered. Determinism and perfect information simplify both the task and the creation of the simulator. The state in Go also contains all history required to perform the next optimal computation - i.e. the system doesn't have to have any notion of memory - yet the state in most real world tasks is far more complicated, at least in regards to what needs to be remembered and how that should be stored.
+ The point of the hardware requirements is that hardware advances will advance the state of the art in Go but will not do the same in many other ML tasks. Whether or not we have 1x or 10x AlphaGo distributed's computing power in our pocket is not the issue - the issue is that such computing power won't assist many tasks as the potential of our ML models are not compute bound.
There's also disagreement about how concerted the "10 year jump" was, which is mentioned in the article by Miles Brundage. Many people (including Michael Bowling, the person who designed the system that "solved" limit heads-up Texas Hold Em) predicted professional level Go play around now. Whilst it might be held by many, I also feel it was a media reinforced estimate.
And about the state and memory concern, mcts does not care directly about it, since for it the simulator is used as a black box and its internals are irrelevant to it. Instead, as long as any environment configuration can be described in terms of state (essentially a unique state->number conversion must be possible - and even then not always) mcts will work. And since it also does not care about the size of the state space, the concern that having memory as one of the factors in the state would be problematic is also unfounded.
I also disagree on the specificity of AlphaGO. Mcts has been used successfully in many fields after its initial usage and tuning for Go. I did my thesis on similar algorithms. In the same way, it does not matter whether AlphaGO can be directly used on other problems. What matters is the new idea of using NNs in order to improve substantially and with little overhead the value estimations used by mcts to explore the decision tree. This is the true breakthrough. The fact that the first implementation of this idea is a Go playing program is irrelevant, it's more like a showcase of the goodness of the approach.
Growth mindset.
And it is very likely that it will never be, the number of combinations is simply too large.
Is that really true? Demis stated that the distributed version of AlphaGo beats a single machine version only 75% of the time. That's still stronger than virtually all human players, and probably would still beat Lee Sedol at least once.
Now was a sweet spot in time. This algorithm ten years ago would likely have been pathetic and in ten years time will likely superhuman in the same manner that chess is.
None of these constitute a general advance in AI however.
[1]: http://www.milesbrundage.com/blog-posts/alphago-and-ai-progr...
"Facebook’s darkfmcts3 is the only version I know of that definitely uses GPUs, and it uses 64 GPUs in the biggest version and 8 CPUs (so, more GPUs than single machine AlphaGo, but fewer CPUs). ... Darkfmcts3 achieved a solid 5d ranking, a 2-3 dan improvement over where it was just a few months earlier..."
By a single machine winning many games relative to the distributed version, really it's just saying that the value/policy network is more important than the monte carlo tree search. The main difference is the number of tree search evaluations you can do; it doesn't seem like they have a more sophisticated model in the parallel version.
This suggests that there are systematic mistakes that the single 8 GPU machine makes compared to the distributed 280 GPU machine, but MCTS can smooth some of the individual mistakes over a bit.
I would suspect that the general Go-playing population of humans do not share some of the systematic mistakes, so you likely won't be able to project these win/loss distributions to playing humans.
But when you're pitching it at a top class opponent in a match getting global coverage, you want all the incremental improvement you can get.
Edit: typo
Stunningly big, or stunningly small?
The CPUs would cost around $65/hour on Google Cloud. I can't immediately find pricing for GPUs on either Amazon or Google, but let's suppose it doubles that price.
It's pretty small potatoes.
Especially if you put it in the context of a project with ~15 researchers.
Yet ... it really doesn't mean almost anything that people are predicting it to mean. Slashdot went so far as to say that "We know now that we don't need any big new breakthroughs to get to true AI". The field of ML/AI is in a fight where people want more science fiction than scientific reality. Science fiction is sexy, sells well, and doesn't require the specifics.
Some of the limitations preventing AlphaGo from being general:
+ Monte Carlo tree search (MCTS) is really effective at Go but not applicable to many other domains we care about. If your problem is in terms of {state, action} pairs and you're able to run simulations to predict outcomes, great, but otherwise, not so much. Go also has the advantage of perfect information (you know the full state of the board) and deterministic simulation (you know with certainty what the state is after action A).
+ The neural networks (NN) were bootstrapped by predicting the next moves in more matches than any individual human has ever seen, let alone played. It then played more against itself (cool!) to improve - but it didn't learn that from scratch. They're aiming to learn this step without the human database but it'll still be very different (read: inefficient) compared to the type of learning a human does.
+ The hardware requirements were stunning (280 GPUs and 1920 CPUs for the largest variant) and were an integral part to how well AlphaGo performed - yet adding hardware won't "solve" most other ML tasks. The computational power primarily helped improve MCTS which roughly equates to "more simulations gets a better solution" (though with NNs to guesstimate an end state instead of having to simulate all the way to an end state themselves)
Again, amazing, interesting, stunning, but not an indication we've reached a key AI milestone.
For a brilliant overview: http://www.milesbrundage.com/blog-posts/alphago-and-ai-progr...
John Langford also put his opinion up at: http://hunch.net/?p=3692542
(note: copied from my Facebook mini-rant inspired by Langford, LeCun, and discussions with ML colleagues in recent days)