AGI will come when the substrate for AGI is laid down. We probably have already done that. As cloud computing matures, we will approach a world where every computer offers its computational resources on a global compute market. At some point between here and there, we will reach a place where compute is cheap enough that experiments will occur regularly that are sufficiently massive to dredge up the solution. And improvements in MRI fidelity, underlying improvements in computing technology and other things will only shorten the fuse. There is no reason why this couldn’t happen tomorrow.
Only one thing is sure: without a computational substrate to stir from, AGI cannot come to be.
His point was mostly that, way before you achieve the kind of AGI portrayed in fiction, you'll have semi-intelligent interdependent systems that cause a lot of trouble due (like the kind that already happens to a lesser degree). Those are the ones that we should worry about right now.
If that growth doesn't cap out before AGI, then at some point we'll harness more computational power than a human brain, in a more controllable way. And around that point AGI probably goes from impossible to possible to easy in a stretch of about 6 years.
edit turns out 6 years was about the timespan for computer Go to span "plays like a competent amateur" to "superhuman". There was barely anything worth reporting before 2010 [0].
What you're describing as hype/greed around badly understood algorithms could easily manifest as 'cult-like oracle religions'.
Combine that sort of thing with the superstitious, wealthy people playing power games, and what you're describing is essentially the exact same thing as the "bad science fiction" described above, just from an arguably more grounded angle.
Using Siri, not itself the outmost edge, but an estimate of the state of the art... If you have an iPhone, take it out and ask Siri how much memory is left on your phone. If she doesn't know how (she doesn't) explain it to her (she can't learn). This is not a philosophical question, it's not a question that requires opinion forming, or fuzzy guesses, it's not a question that requires establishing ad hoc priors that update in real time, it doesn't need to parse an idiom, or draw connections, or to disambiguate complex meanings. It's a simple request, and a simple task. To anyone overly paranoid about the AI issue, this might actually help attenuate those fears when someone starts going on about the singularity being upon us... you just need to take out your phone and ask Siri how much free memory you have left; if she tells you... panic.
I am not worried about AGI. We're nowhere near it. We don't even have a good idea what it would look like.
I am worried about people succumbing to hype or greed and using badly understood algorithms to control critical infrastructure. We already have a preview of what that can look like with Facebook's and Google's content filtering and recommendation algorithms aiming for ever higher "engagement". It's not pretty. Other examples include HFT bots and Amazon's pricing bots. It's funny to see $10,000 book on sale. It's less funny to see a flash stock market crash. It will be totally not funny if something like that will create a global economic crisis through some subtle yet "wide" feedback loop no one is aware of.