If top humans get beat 5-0 with significant handicaps then it is likely AlphaGo is superhuman. However, it is expensive to run AlphaGo so it is unlikely that we will know the true strength of AlphaGo for a while (more challenges) or until hardware catches up.
Update: typos and clarifications
Ok flippancy aside, there are two problems that make techniques like this single-domain: network design and network training.
The design, uses multiple networks for different goals: board eval (what boards look good) and policy (which moves to focus on). Those two goals, eval and policy, are very specific to go. Just like category layers are specific to vision and LSTM is specific to sequence learning.
Network training is obviously hugely resource intensive -- and each significantly complex problem would need such intensity.
It is amazing the variety of problems DNNs have been able to do well in. However, the problem of network design and efficient training are significant barriers to generalization.
When network design can be addressed algorithmically I think we may have an AGI. However, that is a significant problem where you automatically add another layer of computational complexity so it is not on the immediate horizon and may be 50+ years down the road.
AlphaGo is rather different in that it actually has a representation of the game of Go and it knows how to play. I don't doubt at all that it's intelligent, in the restricted domain it operates in. But I do doubt that it's possible for an intelligence built by humans to be "superhuman" and I don't see how your one-liner addresses that.
Why would an intelligence built by humans not be able to be superhuman? The generally accepted definition seems to be "having better than human performance" in which case it seems we've done it many times (like with calculators).
I don't think there's a generally accepted definition and I don't agree that performance on its own is a good measure. Humans are certainly not as good at mechanical tasks as machines are -duh. But how can you call "superhuman" something that doesn't even know what it's doing, even as it's doing it faster and more accurately than us?
Take arithmetic again. We know that cat's can't do arithmetic, because they don't understand numbers, so it's safe to say humans have super-feline arithmetic ability. But then, how is a pocket calculator super-human, if it doesn't know what numbers are for, any more than a cat does? There's something missing from the definition and therefore the measurement of the task.
I don't claim to have this missing something, mind you.
>> Why would an intelligence built by humans not be able to be superhuman?
Ah. Apologies, I got carried away a bit there. I meant to discuss how I doubt we can create superhuman intelligence using machine learing specifically. My thinking goes like this: we train machine learning algorithms using examples; to train an algorithm to exhibit superhuman intelligence we'd need examples of superhuman intelligence; we can't produce such examples because our intelligence is merely human; therefore we can't train a superhuman intelligence.
I also doubt that we can create a superhuman intelligence in any other way, at least intentionally, or that we would be able to recognise one if we created it by chance, but I'm not prepared to argue this. Again, sorry about that.
Hm. Strictly speaking I believe my pocket calculator has an FPGA, a general-purpose architecture that in my calculator happens to be programmed for arithmetic, specifically. So I think it's accurate for me to say that, although the calculator has a program and that program certainly is a representation of arithmetic, I have to provide the interpretation of the program and reify the representation as arithmetic.
In other words, the program is a representation of arithmetic to me, not to the calculator. The calculator might as well be programmed to randomly beep, and it wouldn't have any way to know the difference.
(But that'd be a cruel thing to do to the poor calculator).
[1] https://en.wikipedia.org/wiki/Harry_HaftTo be honest- I don't have one. My intuition is that we can't have a good definition of "superhuman intelligence" because having one would require us to demonstrate superhuman intelligence ourselves. Which is obviously a contradiction.
So I think the word "superhuman" must imply a fair competition, in the sense that the participants are competing using comparable approaches. For some definition of comparable.
I still don't agree that this is the case, and I don't care what a thousand Google-hyped press releases say, beating the best human player in anything is not "superhuman" and "superhuman" performance has not been achieved by anything yet. [1]
Why do I think so? Two reasons.
One, because you can be entirely human and still beat all other humans, without fail for a very long time. Long "winning streaks" in professional sports are very well documented. For instance Rocky Marciano went entirely undefeated in his whole heavyweight boxing career. In chess, Mikhail Tal went undefeated for 95 games. And so on, so forth.
Of course most humans' winning streaks end eventually. That's because our performance degrades over time. When a computer wins against the best player, it keeps on winning, and in fact it actually gets even better over time.
Still, and this is number two: in most instances where a computer does better than a human, we don't claim superhuman performance. Automatic calculators, going back to mechanical calculators, have been better than humans at arithmetic for a very, very long time. I would wager that nobody discusses pocket calculators as exhibiting "superhuman" performance. You only hear this sort of claim when it comes to Deep Blue, AlphaGo or Watson.
So maybe we need a better definition of what it means to be "superhuman" that covers both pocket calculators and AlphaGo. Without one, I don't accept that the performance of AlphaGo can be said to be superhuman, unless pocket calculators' performance is also celebrated as superhuman.
_______________________
[1] I'm perfectly willing to go even further than that and say that we can't make machines that have superhuman intelligence and that even if we did, we wouldn't be able to recognise them (this last bit is similar to what the GP says).