I'm pretty sure I heard them say this.

They can see certain stats and high level overview.

But they have no idea what it's thinking.

They know the general pattern of the algorithm. They even explain it.

But the algorithm involves two deep neural networks, and they don't really know what's going on inside them.

One of the developers showed up during the commentary on the second game and talk about this stuff:

https://youtu.be/l-GsfyVCBu0?t=2519

AlphaGo developers don't understand how it works in the same way you wouldn't understand how the program you've written to find prime numbers actually found a big prime number. The sequence of operations is known, but numbers are too big to be comprehended.

I think its more like, real parents dont understand why their children do the bizarre things that they do.

I don't think so.

Which kind of makes me think. Would machine learning succeed even mildly at recognizing primes? Would we be able to decode the final weights after weeks of learning, and find a sieve program encoded as data?

Well apparently my question leads to some deep mathematic theories about languages encoded by data. http://cstheory.stackexchange.com/questions/15039/why-can-ma...