Even though they are high-profile people, in DL, since people still don't know a lot about it, their confidence means nothing.

When I was in college, my professors/textbook alike, claimed that in order to conquer Go, we probably needs quantum computer or something really sci-fi, maybe in 50 years, even 100 years. Guess what, no one, not even the most optimistic person, would predict it will beat the best human player in 10 years. So, yeah, AGI might happen, maybe in 10 years, maybe in 100 years, but until it happens, no one really knows when is that moment exactly.

This is what I call living in a bubble, dear sir/madam.

I grew in a small town in Eastern Europe (~100k population) and me plus the local 50 or so programmers (using Apple II and 8086-based computers), in 1996, while we were in our teenage years, were painfully aware that games like chess and Go are brute-force-able with some elements of recognizing and memorizing viable strategies (and throwing out the unviable ones early on). And that was what, 15-19 year old late teenagers, and we knew it. So be sure that many people knew it -- they just never chose that area of expertise specifically.

This also says a lot about the quality of the average professors and textbooks, but let's not go off-topic, plus it's a huge discussion area.

Once someone builds it, stopping it might be very difficult. Here's why: https://youtu.be/4l7Is6vOAOA (less than 9 minutes and very clearly explained).

Dismissing even a 10% chance of possible catastrophic risks is not what we practice in any other domains. Would you dismiss a concern over airplanes that have a 10% or even just a 1% chance of crashing?

Certain 30s physicists were making "My greatest fear is that my research is too successful" warnings that turned out to be pretty on-point.

I think the best way to understand AI tech is to try it out yourself. The closer you are to understanding its internals, the more informed you'll be. Fundamentally, it's currently all statistics. It's up to you, then, to decide whether humans are just statistical machines, or if we have something more that may not be understood by science yet, such as free will.

So the reality is, it is like ancient human knows birds can fly with wings, but people can only fly in their dreams. Now, we know people can think intelligently with their brain, but no one knows how to make computer works the same. It is far too early to talk about the risk, until we have the Wright brothers of AI to enlighten us on such possibility.

Also, the goalpost for identifying something as a challenging cognitive skill worthy of the name AI is moved almost every time we make progress so people keep denying that we are closer to AGI. AlphaGo is the latest example.

Some AI researchers and CS people believe that Natural Language Understanding (NLU) is AI-complete, i.e. once we solve it we basically solve AI in the sense of AGI. I do not personally believe that--There are certain human cognitive skills that are not required for NLU. But I do think that solving NLU does bring us closer to solving AGI.

Let's say someone makes progress on NLU. What would be the minimum level sufficient to convince you that AGI is possible? Why minimum? Because we don't want AGI to be right at our doors before starting to prepare.

* Would getting 80% on Winograd Schema be sufficient?

Other suggestions are welcomed, including by others.

https://en.wikipedia.org/wiki/Winograd_Schema_Challenge

This itself presents several big challenges: how do we represent the concept of reasoning, in mathematical form? Knowledge required or not? What is the representation of such knowledge?

A truly intelligent machine, should be able to take what human takes, a piece of text, then build its knowledge base from there, then answer a question just like human, then give explanation when asked for it.

There's a small chance aliens could find us, too. Yet, we don't know any more about that than we do about AI.

The most imminent threat to humans is humans. In the nearer future, it seems more likely that someone will use machine learning technology in weaponry to cause harm than create AGI. Or, that humans will undergo a global "cultural revolution" like in China or Cambodia (or ISIS now), killing anyone with an education, under the guise that "machines are evil", yet with the true goal of obtaining power, as humans often do.

Ultimately, people are free to work on what they want. If some people want to work on AGI safety, that's great. Perhaps their goal is to make humanity a little more kind. That's certainly worthy.

For me, I'm more interested in seeing research focused on giving better tools to doctors for diagnoses. Radiology is a great place to apply image recognition systems. We just need some good labeled datasets.

Sometimes I get frustrated by the "AGI is an existential risk" quotes like Musk's because I feel that's a distraction from some really beneficial applications of machine learning that will help save lives. It could cause over-hype, leading to another AI winter, pushing back life-saving advances 15 years, as happened after the 80s.

Personally I don't have a problem with the kinds of things some AGI safety researchers, like Stuart Russel, says about AGI. He isn't saying it will arrive in 2030-2040, like Musk.

Musk has an incentive to capitalize on people's fear of AGI. It attracts investment in his non-profit, which can in turn help him continue building his in-house AI program at Tesla. Primarily I disagree with his conclusions about AI tech, but I'm also weary of his motivations. I think some people view him as altruistic and I just don't see that.

> What would be the minimum level sufficient to convince you that AGI is possible?

A system that can determine its own goals. We don't have anything close to that yet, and I can't see anything short of that being a good indicator that we are close to AGI.

Even if an AI does not determine its own ultimate goal(s), it could still cause us a lot harm by creating subgoals that do not align with human values, which no one can articulate in full yet.

Overall, the controversy is worth it because the force for progress is strong enough that another AI winter is highly unlikely if useful applications continue to be developed. Without the warnings, research on Provably Safe AI might not have sufficient resources especially talented people working on it.

Really disagree here. All it takes is a few well positioned people, like those at Nnaisense, to create companies promising AGI and sucking up investment dollars. We all know greed is rampant in finance, and AI tech is a big word in startups/innovation these days that can land big VC dollars. There is corruption/snake oil in tech too, and you'll find the greediest float around the most hyped tech.

I think companies like Nnaisense are the wrong place to invest, and that there is much more practical work to be done to advance humanity.

> Sometimes I get frustrated by the "AGI is an existential risk" quotes like Musk's because I feel that's a distraction from some really beneficial applications of machine learning that will help save lives.

While I absolutely and adamantly disagree and I can't ever agree that warnings must be dismissed as "distractions", I extract from your statement here that you want us to move forward to a true AI no matter what. But then...

> ...and that there is much more practical work to be done to advance humanity.

Right now AI looks like snake oil selling, you realize that, right? I stand fully behind the statement that we have tons of practical and increasingly urgent issues to solve here in the material Earth.

Unless of course you're implying that the strife for inventing a true AI (=AGI) is a practical work. IMO it isn't. It's better than the medieval alchemy, but not by much.

Warnings can be faked. Fear mongering can be lessened when people are informed on a subject.

> I extract from your statement here that you want us to move forward to a true AI no matter what.

Where in the devil did you get that? I said no such thing. I think investments towards building AGI are largely a waste of money. If people want to spend their money on that, that's fine. I won't.

> Right now AI looks like snake oil selling, you realize that, right?

AGI does, yes. And, for those who do not understand the difference between data science and AGI, then probably all "AI tech" seems like snake oil. But it isn't. It's already helping doctors detect cancer, among other major issues.

> Unless of course you're implying that the strife for inventing a true AI (=AGI) is a practical work

I'm not. I have no idea where you got that from my comments.

Work is being done on climate change, but at the current rate, we're way off preventing catastrophic effects. I'd say that our current level of effective effort basically amounts to ignoring the problem.

There is a non-negligible possibility that AGI will be invented in 30 years. Many AI experts agree on that. A number of experts also believe that its invention could be highly beneficial or catastrophic, depending on its form and our preparation.

With cost-benefit analysis based on the best knowledge we have weighed by probabilities, it is clear that AGI risks are more substantial and worth at least as much investment as climate change. The current funding for AI Safety Research is not even 1/10th, perhaps less than 1/100th, of climate change funding. Inaction is also an action.

If you do not trust intelligent domain experts, and also intelligent non-experts with almost no conflict of interests like Bill Gates and Stephen Hawking, then please let us know which source(s) of knowledge we should rely on instead.

Note: I believe we should fund both. A certain but slow train wreck and an uncertain but even more catastrophic and possibly speedier train wreck are both worth preventing.

That's not true. Arctic methane release has the potential to become devastating within a small number of decades. From [1]: Shakhova et al. ... conclude that "release of up to 50 Gt of predicted amount of hydrate storage [is] highly possible for abrupt release at any time". That would increase the methane content of the planet's atmosphere by a factor of twelve. That would be catastrophic by anyone's measure.

Also: In 2008 the United States Department of Energy National Laboratory system identified potential clathrate destabilization in the Arctic as one of the most serious scenarios for abrupt climate change.

[2] is a video from the Lima UN Climate Change Conference discussing this. The people talking about this are certainly not "no-one serious". See 34:23 for Ira Leifer, an Atmospheric Scientist at the University of California, saying that 4 degrees of warming means the Earth can probably sustain "a few thousand people, clustered at the poles".

I see 4 degrees of warming being bandied about a lot these days, but very little discussion of how catastrophic that would be. It also seems a lot more likely than AGI becoming a problem, to me at least. Sure, we should fund research into both but only one of them has me worried about the world my daughter will inherit - one is basically purely hypothetical right now.

    [1] https://en.wikipedia.org/wiki/Arctic_methane_emissions
    [2] https://www.youtube.com/watch?v=FPdc75epOEw

Fighting a passive adversary is a much easier thing than fighting an active one (especially one smarter than you).

It's also at odds with the study that has the very biased question asking experts "when do you think AGI will arrive?". Many chose not to answer the question, and thus their answers can't be averaged in.

Many of the AGI safety conscious crowd skip over the question of whether we will be able to create AGI and just assume we will.

There's no precedent for this kind of discovery. Fire, the lightbulb, and nuclear weapons all pale in comparison.

Nuclear weapons were deemed impossible by some experts at the time. Throughout history, whenever there are a number of experts who believe something to be impossible and a substantial number who believe it to be possible, the latter are almost always right.

See also my other answer that begins with "Unlike airplanes, ..."

Sure, go for it. I think this has been in the news a lot recently because Musk told government leaders to expect AGI as early as 2030-2040.

It's a bit of a ridiculous claim. One of the biggest AI competitions, ImageNet, expects to run for the next 12 years, and that's just for making sense of images.

Perhaps it'll be done in less than 12, however, the predictions of the ML community and Musk do not line up.

> Nuclear weapons were deemed impossible by some experts at the time.

I suppose time will tell with regards to AGI then. I suspect we'll have another economic dip before 2030, and all this hype will die down before then. And, we'll remember this time as any other in the past when we predicted flying cars by the year 2000

Edit: I think the announcement above is about object recognition on still images, which is largely solved. The "new ImageNet" one on Kaggle in the reply below is for videos.

I don't really understand the press release on image-net.org. Maybe they meant the last competition hosted by the same group?

Edit: See page 84 of the PDF linked here: https://www.google.com/search?q=kaggle+site%3Aimage-net.org

Ps- screw you Google for not letting me copy links to PDF search results via my phone

Ok, will try

> Specifically I wonder what would be a minimum threshold for you to believe that AGI is likely possible.

Current machine learning tech doesn't have any sense of creating its own goals. We also don't know how to encode that. All of the systems we have target specific problems, like playing Atari games, identifying objects in images, or playing Go. These are all great milestones in machine learning tech. They do not indicate we're much closer to developing a new intelligence. If we could set a system out there and have it determine its own goals and learn things on its own, then maybe I'd believe we're close to creating AGI. As it stands, we don't know how to run a program that decides what it wants to learn. Every program we write has our hand in it.

My suspicion is that our quest for developing AI leads us to discover more about ourselves than AI itself. I think we'll continue to refine our definition of intelligence, and continue to use the tools we build to augment our own intelligence.

Most humans do not really come up with our goals on our own either. We are heavily influenced by genes and environment. People's personality are partially changed by the environment and occupation they are in. Twins adoption studies also show that genes have surprisingly strong effects on personality even after decades living apart.

By your definition, humans do not possess general intelligence.

Edit-Reply:

I don't think we can resolve an age-old philosophical question on Free Will in a comment thread, so I'll leave it at that. :) However, General Intelligence and Free Will are largely orthogonal.

We can imagine an alien who is highly capable of any tasks human can perform and beyond, including coming up with creative subgoals as necessary, but will only serve their master's ultimate goal(s). An AGI could be like that alien but built to serve human masters. The twist is that when someone orders an AGI to efficiently reduce animal suffering, without value alignment with humans, it could come up with euthanasia as a subgoal to reduce the suffering of starving kittens.

I disagree. I believe we have free will, in addition to being influenced by our environment/genes.

This is my general point, that contemplating AI is a way for us to investigate how we ourselves operate. You have your theory and I have mine. Nobody has shown how we can be replicated.

EDIT

> We can imagine an alien who is highly capable of any tasks human can perform and beyond, including coming up with creative subgoals as necessary, but will only serve their master's ultimate goal(s). An AGI could be like that alien but built to serve human masters. The twist is that, without value alignment with humans, some subgoal could be killing starving kittens to reduce their suffering

If the AGI's goals are determined by a human, then it is technically trivial to encode human values. The hard part is convincing humans not to encode evil values.

So it isn't that AGI would go rogue, it's that humans could. That's an age-old, Adam & Eve question. It's not specific to AI technology.

You might have heard of the Trolley Problem. There are many and more complicated variations of that.

This Harvard session on Justice (The Moral Side of Murder) is edifying and surprisingly fun to watch: https://www.youtube.com/watch?v=kBdfcR-8hEY

The problem with AGI is more acute than most other technologies because it is software-based, which makes it nigh impossible to regulate effectively, especially globally.

Don't kill humans sounds like a good start.

Honestly, this problem seems a lot simpler to me than creating AGI itself.

I agree it is challenging. The most imminent question is probably, "should a self driving car be allowed to kill 10 people on the sidewalk to avoid a head-on collision, or should it accept the head-on and let the driver die?"

These domain specific problems will come up before the AGI one does, and we can address them as they become relevant.

You've noted that regulating AGI is difficult, and it sounds like you don't have any other solution in mind.

It is definitely very challenging to create one, and more challenging than creating a random AGI. The morality also needs to be integrated into the Safe AGI as its core that is not susceptible to self-modification ability that an AGI could have. Thus, we need to work on that aspect of AGI now.

Stuart Russell has outlined his preliminary thought on the topic: https://www.ted.com/talks/stuart_russell_how_ai_might_make_u...

> So what would be your equivalent position in the 1940s about the nuclear solution?

I've seen people throwing that example in this thread and I don't see why.

Multiple countries invested heavily in building the bomb. It was not a question if building a bomb was possible, but rather how soon, and by whom.

> In other words, what are the risks and benefits of adopting a "worried" versus a "don't worry about it" position?

Some people do worry about it, but everyone need not. I don't.

> If there's a possible huge meteor (say 30% chance) of hitting the Earth in about 50 years, would you be "worried"?

Good example. No I wouldn't because I know we have systems monitoring for such an event. If something is coming, we will know about it. I'm not sure how far in advance we would know, but I expect as it got nearer more and more people would do their best to come up with a good solution.

We're no nearer to AGI today than we were when neural networks were developed 30 years ago. Science fiction would have you believe it, but we don't look at SciFi movies of the 50s as particularly prescient, and we shouldn't do so for today's films either.