Current AI technologies give us no insight about the probability of an AGI

Because that is exactly the point. We can't look at what we have built today and extrapolate that AGI will come from it.

All anyone can do is speculate wildly on AGI just as they were able to in 1956 [1]. The difference is that we see automation/computing as ubiquitous (smartphones/computers in 2015), rather than around the edges (Radio/TV 1956). Which brings it closer to home in some ways.

[1] I chose 1956 because of the dartmouth conference on AI

Saying that we don't have to worry because AI development will take 1000 years is also missing the point. Research into unsafe AI is inherently faster than research into safe AI, because the latter problem is more constrained. If unsafe AI kills us all, the best time to start working on safe AI is 100 years ago, and the second best time is now.

Saying that we don't have to worry because smarter than human intelligences will have a better understanding of morality is also missing the point. All currently known mathematical formalisms for decision-making allow you to plug in arbitrary utility functions, including "immoral" ones. Thinking that future decision-making formalisms will support only "moral" utility functions is extreme wishful thinking.

Saying that we don't have to worry because brain emulations or human cognitive enhancement will come first is also missing the point. These things can be just as dangerous as AI (e.g. see Robin Hanson's descriptions of how mind uploading increases Darwinian pressures on everyone, including non-uploaded people), and can also lead to AI more quickly.

And so on.

If you want to feel safe and protected, no one can stop you. But you don't have any good arguments for feeling safe. I'm sorry, but you really don't. I've been on the lookout for such arguments for years, and I'd be happier if they existed.

I mean that is what I think, and most of the community thinks, but we haven't proven that empirically - nor could we, its unprovable.

But you don't have any good arguments for feeling safe.

Correct. Notice that I am not saying that we are safe. I personally think we aren't safe [1], but that is a different discussion which is equally as baseless.

The key here is that the argument that AGI will not be safe, and therefore needs to be regulated to make it safe, not only misses the point but ensures that we would never make one. The current level of funding for AGI development and understanding of the roadmap for AGI is absolutely dismal given it's potential impact. If we now add the constraint that in order to make one, we also ensure it doesn't threaten humanity - then you have sounded the death knell for AGI. Once we make AGI - in the sense that it is a recursively self improving system with human equivalent capabilities across domains - there is no way we will know what will happen. That's it.

[1] Which is why I am personally in the transhumanist camp.

I'm going to play Devil's Advocate on this.

While it is true that all known decision-making formulas allow for immoral utility functions, I think it's a flaw of current approaches, such as reinforcement learning, that they only allow nonspecific utility functions. I can't write down an AI program to maximize paperclips -- there's no way, in current formalisms, to specify what that means.

I think the FAI ideologues and the mainstream machine-learning community can both rally around the prospect of being able to specify a goal in terms other than reinforcement. Scientifically, that's an entirely sensible research question for the field of AI/ML to tackle, and it also constitutes a major step away from the scenarios in which a dangerous AGI winds up accidentally programmed with random, arbitrary goals that nobody actually wanted. After all, fundamental philosophical questions aside, a machine that flies out of control at random when turned on is simply a bad machine.

That said, it's indeed very hard to directly specify a utility function involving paperclips. If our universe were a Game of Life universe and we knew exactly which configuration corresponds to a paperclip, I'd be able to do that right now. But since we don't know the true laws of physics, the "hard way" involves encoding some kind of Solomonoff prior over all possible physical universes, and a rule for recognizing paperclips in each of them. That's kind of a tall order.

There's a shortcut that involves encoding a precise mathematical description of a human mind into an AI, and passing the buck by saying "the AI must maximize whatever function the human mind would output, given enough time to think". That would actually be straightforward to implement, if we had a description of a human mind that we were willing to trust. Unfortunately, the naive form of that approach immediately fails due to acausal blackmail. There's no obvious fix, but some folks are trying to devise non-obvious fixes, and I suppose it can be made to work in the long run.

Of course, after you can formulate a utility function in terms of a precise description of a human mind, the next step is reformulating it in terms of the outputs of an actually existing person, say some webcam videos and written instructions. You instruct the AI to infer the simplest program that would generate these outputs (using something like the Solomonoff prior again), and then proceed as in the previous step. That part has its own problems, but I expect that it can also be made to work.

The whole approach is kind of a long shot, but I hope that I've given a sense that the problem could be solvable by careful human effort on the scale of years. There are other approaches as well.

Or we just need an approach to AGI that gives us more conceptual abstraction than AIXI and its Solomonoff-based reasoning. Which we needed anyway, since AIXI_{tl} is "asymptotically optimal" with an additive constant larger than the remaining lifetime of the Earth.

Luckily, there's quite a large amount of bleeding-edge research into exactly that: learning "white-box" representations that are understandable and manipulable to human operators.

And I'm very skeptical that mathematical Platonism is useful for AI: "mathematical objects directly" do not exist in the real world, and it is very much real-world things on which we want our software to operate. "Conceptual abstraction" simply refers to a learning algorithm that possesses a representation of, for instance, a chair, that is not composed entirely of a concrete feature-set (visual edges, orientations, and colors) and can thus be deployed to generatively model chairs in general.

Computational cognitive science is working towards this sort of thing, and the results should start to hit the machine-learning community fairly soon.

Well, there's an influential minority that thinks mathematical objects are all that exists (Tegmark multiverse). I don't necessarily agree with them, but that's one way to rigorously define the domain for a utility function, in a way that is not obviously exploitable. Another way is to define utility in terms of an agent's perceptions, but that is exploitable by wireheading, and IMO that flaw is unfixable as the agent gets more powerful. I'm not aware of any other approaches that are different in principle from those two, so I'll stick with the lesser evil for now, and hope that someone comes up with a better idea.

I do agree that we lack evidence of a broad community of AI experts who have been able to make good predictions, which potentially puts AI research even behind economists! Nonetheless, we'd expect there to be a period between "wild guessing" and "good and reliable models"; Gates and Hawking either think we're in that middle ground, or they think the risks are considerable enough that it's worth pushing towards it now.

If I apply your analogy correctly, then warning about AI now, is the same as it would be warning about global warming in the 19th century. Not very logical and certainly very paranoid.

I don't see what's irrational about trying to solve problems before they become problems, rather than trying to do damage control after the fact.

Nothing in the world is more dangerous than premature attempts at safety.

But are you saying there's absolutely nothing that could've been done better in the industrial revolution? With some foresight we might have thought to develop solar power more urgently, for example, but no one thought we needed to. Worrying just a little might have changed that.

The point is not to stop progress, just to approach with caution and be mindful of what the long-term implications are.

And that, mind you, is still with the anachronistic application of 21st-century knowledge to the matter. If people in the 19th century had actually tried to figure out what they should be worrying about if they were going to worry two centuries prematurely, they would likely have come up with something completely different.

With some foresight, people were trying to develop solar, nuclear, and fusion all the way back in the '60s and '70s, but political interests ensured that research funding was reallocated towards shorter-term political projects.

It's akin to someone back then worrying about the long-term effects of steam/water vapor.

You don't know if vapor is actually a problem, or will become one, and you don't have any data in any field to back up your concerns.

When pharma companies produce a new drug which appears to have a positive effect, they don't know that it will cause problems and don't have any data to back up any concerns.

So do they rush full-steam-ahead into the unknown? Do you unleash the drug to whomever wants it, without regulation? No, of course not; you approach with caution, you hold clinical trials, you collect data as you go and you look for potential problems before the drug is widespread.

Arguably a similarly controlled approach would have helped in the case of carbon output and will help in the case of AI. It's not about fear mongering or preventing progress, it's about discussing ways to approach with caution and solve problems (and minimise the damage they cause), if and when they arise.

If someone can do forecasting of AGI with similar empiricism that would be revolutionary and welcomed.

Moravec's 70s papers are roughly comparable to Arrhenius's formula. The basic computational power predictions (a minor extension of Moore's law) still seem to be basically correct. There's a reasonable debate to be had about whether more interesting predictions made by Moravec/Kurzweil/etc. have been validated or not.

But AI is plausibly 50 years away from harming large numbers of people[1]. By that standard of "years until harm", global warming should have been first publicly discussed in the 1980s or 90s. That is a bit later than it happened, and later than some would like.

[1] If researchers develop models that say it's more like 100 or 500 years out, I think that would be great and very helpful. But several researchers have put out credible models which put a crisis in the 40s to 60s. Skepticism about their approaches that identifies specific flaws or produces improved models is good, much better than hand-waved dismissals.