News for hackers. Stuff that matters.
Re:One Giant Step for a Chess-Playing Machine by mindgam3 on Wednesday December 26, 2018 @09:01PM (#18766657)
Is it just my sonar beeping off the charts, or does anyone else hear the unmistakable signs of a submarine article? (1)

Apologies in advance for what may be perceived as a rant. I have a very low tolerance for clickbait-y BS like this as it pertains to my own passions as a lifelong chess devotee and former professional player.

First, the author of the article has no professional credibility in either chess or machine learning. He's a professor of math and a writer. No disrespect to either math or writing, I love and value both very highly, but they have very little to do with chess and machine learning per se.

The problems is he tries to present AlphaZero as "humankind’s first glimpse of an awesome new kind of intelligence," which is really a bit of a stretch unless you add the disclaimer that technically all AlphaZero does is play 3 types of perfect-information games quite well. This is undoubtedly a great accomplishment, particularly in the field of Go which many domain experts felt intuitively would not crack to our AI overlords before another 5-10 years of computing power/hardware advances at least.

(As someone who had the unfortunate label of "prodigy" applied in my youth due to earning the title of chess master at age 10, I consider myself somewhat of a domain expert in chess, and I was one of those people who got it wrong. I barely know the rules of Go, but intuitively I could comprehend that it was several orders of magnitude more complex than chess, and I was really hoping that the Go gurus would fend off the machines for longer. They didn’t. Hats off to DeepMind.)

But. With all due respect to DeepMind engineers for an impressive result in chess and go, it's a bit too early to start thinking of AlphaXXX as an "oracle" where all we can do is "sit at its feet and listen intently" while we would "not understand why the oracle was always right" and eventually be left "gaping in wonder and confusion."

(As an aside, the amount of pseudo-religious worship language in the piece is truly off the charts. I realize it stokes the passions, but it would be great if we could talk about AI’s true strengths and limitations without resorting to such histrionics. But I digress.)

Why is it too early to start bowing down to a new god? Well, for starters, they basically just brute forced the game of Go a bunch of years earlier than predicted, but this wasn't just a pure software win, this was also heavily connected to massive increases in computing power aka GPUs and ginormous cloud-based render farms.

Secondly, the author tries to make the leap from AlphaZero [good at 3 perfect-information games: chess, go and shogi] to what he calls "a more general problem-solving algorithm; call it AlphaInfinity". Note how he invokes the holy grail of AGI (Artificial General Intelligence) without actually using this term, which would set off alarm bells in, well, anyone who knew anything about AI who wasn't employed by DeepMind/Google.

Notice further how this massive leap from "machine that can play 3 games well" to "machine that can, you know, actually think about stuff like a human can, including these pesky 'edge cases' and un-trained-for scenarios that always confuse our algorithms despite their otherwise inhuman level of perfection".

One great example of such a case that may cause one to question these glorious predictions is a research paper titled “Neural Networks Are Easily Fooled by Strange Poses of Familiar Objects” which shows how ML models consistently mistake a school-bus for a snowplow under the right (snowy) conditions (2). Far be it from me to dare bursting the bubble/reality distortion field of certain ML leaders and visionaries, but c’mon - a human child, once they truly learned how to recognize a schoolbus, would never mistake it for a snowplow, even if it was upside down.

This flaw doesn’t mean that we can’t update training data to handle these types of rotations, but it does mean that we have a lot of work to do before we can say that these ML models have in some way grasped the “essence” of “school-bus” or [insert-other-object] here in a deep symbolic way, and by "deep symbolic way" I mean "any way that a human child learns how to do reasonably quickly before moving on to other, exponentially harder tasks".

I could go on, but I won’t. Just in case my overall point isn’t clear:

1. AlphaZero is an unbelievably impressive accomplishment within the limited subset of life that is [chess, go, shogi]

2. ML approaches, even in computer vision, have a long way to go before anything remotely resembling child-level human intelligence

3. Therefore, can we please please stop the marketing masquerading as news articles about DeepMind’s latest result. And if anyone at DeepMind is listening: your product is pretty sweet! It would be better strategically to simply let it speak for itself, without trying to frame it as AGI.

1. http://www.paulgraham.com/submarine.html

2. https://arxiv.org/pdf/1811.11553.pdf

Re:One Giant Step for a Chess-Playing Machine by taneq on Wednesday December 26, 2018 @09:44PM (#18766838)
> technically all AlphaZero does is play 3 types of perfect-information games quite well

You might be right about it being a submarine article but this seems to be underselling it. AlphaZero is (as I understand it) undisputed world champion by an indeterminate margin on the two most popular games, and has become so after only being given the basic game rules. If you told someone from 2015 that this would happen in the next decade, they'd laugh at you.

Re:One Giant Step for a Chess-Playing Machine by mindgam3 on Wednesday December 26, 2018 @09:56PM (#18766888)
As I wrote near the top of my comment, I already laugh at myself for falsely predicting AlphaGo/AlphaZero dominance by a decade, give or take. But the leap from where we are now to AGI is several orders of magnitude more complicated, at least, than the leap from Chess to Go.
Re:One Giant Step for a Chess-Playing Machine by taneq on Thursday December 27, 2018 @12:18AM (#18767521)
I think the issue here is that we started with "traditional" AI (logic solvers, inductive reasoning, graph search etc.) which were very clearly constrained and single-purpose. The goal has always been AGI which is all-singing-all-dancing post-Singularity magic. Now we have taken another couple of steps along the path from AI to AGI, we're gonna need another term for AI which can be applied to a range of tasks without significant further human intervention, but doesn't have the implied general reasoning and perception that our magic AGI will eventually have.

In spaceship terms, what we had was Soyuz and what we wanted was the Enterprise. Now we're about to get a BFR / BFS which can go to Mars and back. Is that a 'real' spaceship?

Re:One Giant Step for a Chess-Playing Machine by cscurmudgeon on Wednesday December 26, 2018 @10:25PM (#18767016)
Re:One Giant Step for a Chess-Playing Machine by slaveofallah93 on Wednesday December 26, 2018 @10:10PM (#18766938)
Is it the undisputed champion of chess? I thought that A0 had never played Stockfish 10, the current best engine.
Re:One Giant Step for a Chess-Playing Machine by cepth on Wednesday December 26, 2018 @11:27PM (#18767303)
FYI earlier this month, they actually did announce the results of a 1000 game match:

https://www.chess.com/news/view/updated-alphazero-crushes-st....

AlphaZero won 155 games, lost 6, and drew 839 games against Stockfish. Granted, this was against Stockfish 9.

This implies that AlphaZero was roughly +52 Elo rating against Stockfish 9 (https://www.3dkingdoms.com/chess/elo.htm).

Stockfish 10 is currently rated ~32 points higher than Stockfish 9 (http://computerchess.org.uk/ccrl/4040/rating_list_all.html). If we were to do very crude transitive reasoning, you'd expect AlphaZero to still beat Stockfish 10.

EDIT:

So apparently the +155, -6 score was against Stockfish 8. Stockfish 8 is rated by the CCRL list at 3379, with Stockfish 10 rated 85 points stronger than 8.

Worth noting that AlphaZero was only given 4 out of 9 hours of total training time when playing against Stockfish 8 (https://chess24.com/en/read/news/alphazero-really-is-that-go...), but I guess we can't make any real conclusions about AlphaZero vs Stockfish 10.

EDIT 2:

So apparently AlphaZero also "defeated" Stockfish 9, but the preprint of the upcoming paper in Science doesn't seem to provide a crosstable.

It seems that Stockfish 8 was given a 44-core machine to play on, and was not constrained in terms of time spent per move etc.

Re:One Giant Step for a Chess-Playing Machine by ubernostrum on Thursday December 27, 2018 @01:31AM (#18767835)
Generally an "undisputed champion" title would require a level of effort that the AlphaZero team apparently isn't interested in. They like to run matches between AlphaZero and Stockfish in private under their own controlled conditions, and publish the results. The rest of the computer chess world likes to run tournaments in full public view, and there are several entities which run tournaments between engines that way which can be used to gauge the relative strength of the engines.

What will be interesting is to see the effect on Leela (https://github.com/LeelaChessZero/lczero), which is public and open-source, takes part in public tournaments, and is built -- as much as possible, based on what's been disclosed in the papers -- to use the same approach as AlphaZero.

Re:One Giant Step for a Chess-Playing Machine by mindgam3 on Wednesday December 26, 2018 @10:34PM (#18767056)
Yeah, fair point. I didn't even think to challenge him on that. Technically it's not undisputed given that DeepMind hasn't made it available for any kind of "fair fight" organized outside of its own office. I'm guessing AlphaZero would still win, assuming DeepMind isn't literally faking data, but the only way to be sure is a match between A0 and SF 10.

FYI, Leela Chess Zero is the open-source project inspired by AlphaZero, and they recently did a match against SF 10, details here: https://lichess.org/blog/XA7juREAAC4AxZsR/deathmatch-leela-v...

Re:One Giant Step for a Chess-Playing Machine by tashi on Thursday December 27, 2018 @01:16AM (#18767762)
Well, he may not have an official rating, but Strogatz did once beat a grandmaster at chess.

http://news.cornell.edu/stories/2006/05/checkmate-professor-...

I've been pretty impressed with him as a sort of cross-disciplinary math wizard ever since I used his Nonlinear Dynamic and Chaos textbook in a class. His research is pretty wide ranging. He may be wrong about the future, but I'd be shocked if it was because of a lack of technical understanding of the algorithms.

Re:One Giant Step for a Chess-Playing Machine by mindgam3 on Thursday December 27, 2018 @01:22AM (#18767793)
You forgot to mention his sole grandmaster win came from a simultaneous exhibition where GM Larry Christiansen played against 35 people at once. Trust me, playing 30 boards at once is a wee bit of a handicap.

Not that I doubt his cross-disciplinary math skills or whatnot. But let's not pretend that he's anything resembling a serious chess player.

Re:One Giant Step for a Chess-Playing Machine by tashi on Thursday December 27, 2018 @03:24AM (#18768267)
I don't think I forgot to mention that, I used a hyperlink to point to it instead. As a guy who loves chess and has studied it off and on over the years, I doubt I'll ever win a game against a grandmaster under any circumstances, so it still seems impressive to me.

But even I know enough, from watching videos on YouTube that analyze some of the AlphaZero vs. Stockfish games, to appreciate that AlphaZero was playing in a style I haven't seen before. What's the threshold for someone serious enough to write a chess article?

Re:One Giant Step for a Chess-Playing Machine by mindgam3 on Thursday December 27, 2018 @02:36PM (#18771905)
Saying "Strogatz did once beat a grandmaster in chess" without specifiying that it was simultaneous exhibition is a lie by omission. I am a chess player, and I would never insult another player by claiming to have beaten him or her without specifying the format. There is a way for you to say what you want to say without exaggerating to make a point.

What's the threshold for someone serious enough to write a chess article?

There's no threshold whatsoever, as long as you communicate a respect for the game and you don't try to use chess as a metaphor to make a larger point about AI and humanity that doesn't hold up to scrutiny. If a rank amateur can appreciate that AlphaZero plays chess beautifully and wants to write about it, by all means. I happen to agree. Just please don't abuse my beloved game for marketing purposes or to win an argument.

Re:One Giant Step for a Chess-Playing Machine by tchainzzz on Wednesday December 26, 2018 @09:28PM (#18766783)
It's funny, articles like this might've inspired me a few years ago to go into this field, but the more that I learn about machine learning techniques, the more skeptical I get when reading things like this. But it's not so much the technical details that bother me but the ethical problem of hype. Hype itself isn't necessarily bad, but throwing it into public at best garners a bunch of harmless clicks and at worst is misleading.

I mean, I guess playing off of the whole "AI apocalypse" and the image of our computer overlords makes a quite evocative point. To be fair, this isn't just a problem with AlphaZero — similar problems in science journalism, especially in the field of medicine (i.e. Immortality is just a few decades away! Stem cells cure cancer!) seem to show up whenever a new or interesting result happens.

Re:One Giant Step for a Chess-Playing Machine by ttonkytonk on Thursday December 27, 2018 @12:28PM (#18770920)
I thought "And when AlphaZero handicapped itself by giving Stockfish ten times more time to think, it still destroyed the brute." was a curious comment. Wouldn't it have been more accurate to say, "And when AlphaZero was handicapped..."?
Re:One Giant Step for a Chess-Playing Machine by mindgam3 on Thursday December 27, 2018 @02:38PM (#18771921)
Yeah. If you look closely, there are a lot of these little rhetorical hacks sprinkled throughout the article which has the effect of glorifying AlphaZero.
Re:One Giant Step for a Chess-Playing Machine by cscurmudgeon on Wednesday December 26, 2018 @10:25PM (#18767012)
You are 100% right. For example, there is zero discussion of General Game Playing (GGP) in these submarine-type articles because the same techniques don't scale at all to GGP.

http://ggp.stanford.edu

Re:One Giant Step for a Chess-Playing Machine by mindgam3 on Wednesday December 26, 2018 @10:37PM (#18767073)
Interesting. I wasn't aware of either GGP or the fact that there was work at Stanford on this topic. Thanks for sharing!
Re:One Giant Step for a Chess-Playing Machine by cscurmudgeon on Thursday December 27, 2018 @11:36AM (#18770498)
You are welcome!
Re:One Giant Step for a Chess-Playing Machine by dirkgently on Thursday December 27, 2018 @09:10PM (#18774372)
> First, the author of the article has no professional credibility in either chess or machine learning. He's a professor of math and a writer. No disrespect to either math or writing, I love and value both very highly, but they have very little to do with chess and machine learning per se.

Well, do you accept Garry Kasparov qualified to comment on this?

"I admit that I was pleased to see that AlphaZero had a dynamic, open style like my own."

"Alpha-Zero is surpassing us in a profound and useful way, a model that may be duplicated on any other task or field where virtual knowledge can be generated."

http://science.sciencemag.org/content/362/6419/1087