Reproductive fitness requires many attributes which obviously compete with intelligence, and evolution is a blind hill-climbing search which cannot easily escape deep local minima.

To me it's obvious that the development of superhuman AI is inevitable assuming we don't wipe ourselves out before we have time to develop it.

Stockfish players chess far better than I can.

AlphaGo plays Go far better than I can.

Random Sudoku bot built on top of minizinc / z3 / whatever SAT solver can solve NP-complete problems better than any human.

--------

Here's the thing: what is "strong AI" ?? As far as I can tell, we're already at superhuman AI in most intelligence tasks (collecting data, collating it together, analyzing it, etc. etc.).

Its called a computer. What if "superhuman AI" is just the stuff we have today at our fingertips? It wasnt very long ago when reading documents and organizing them was considered AI. But today, we call that Google.

Something that could in one package:

- hold a conversation about what it is that needs to be done

- clean a room

- fix a sink

- make a chair or table out of wood

- dig a hole

We don't know how to make AI that will learn to play even simple game like montezuma's revenge in some reasonable play time like 30 min.

We are currently closer to months or years of gameplay required.

Explain to me why a Tool-assisted Speedrun doesn't count as "AI". Its certainly not a human who is playing that (https://www.youtube.com/watch?v=pFB0kuPx5mQ).

> Something that could in one package:

I'm not sure if that sort of thing is worth the investment.

But if we have "Fly the F35 airframe" (which is incredibly unstable), then we suddenly reach "only computers can fly this thing" levels. Just the assumption of super-human control systems is baked into modern products. https://www.youtube.com/watch?v=cyN-CRNrb3E

Our cell phones are largely manufactured by machines these days. A few tidbits here and there remain in human hands for cheapness... but a lot of today's stuff is basically an AI. https://www.youtube.com/watch?v=SRu02F6AOmg

Because TAS are made by human speedrunners, who go through the painstaking process of specifying inputs for each frame. I'm not sure what you're trying to say here, TAS are basically the exact opposite of AI.

> I'm not sure if that sort of thing...

> But if we have "Fly the F35 airframe"...

> Our cell phones are largely manufactured by machines...

That's a red herring. There's plenty of evidence we can make very useful narrow AI in many fields, and nobody is arguing against that.

The point is, they are, well, narrow. What your parent is saying is that there's no evidence that the statistical techniques behind the current wave of AI are capable of anything else.

To paraphrase Norvig, it could well be that we're trying to get to the moon by climbing a tree, reporting steady progress all the way to the top of the tree.

I thought that the problems solved by Bethe, Feynman and their cohort were the easy problems and the current generation is struggling with problems that were much harder, possibly impossible to solve.

One example is the problem of chaotic dynamics with six degrees of freedom as opposed to four degrees of freedom. With four degrees of freedom you have unbroken KAM tori that partition the phase space into areas that it will stay in forever. In 6 DOF or more, some tori are still unbroken but it's possible for the system to go around them, so even in a phase space region that has a lot of intact tori (e.g. the motion of the planets around the Sun) you can't prove anything for the long term (e.g. the Earth doesn't get ejected from the solar system.)

Look also at examples such as the halting problem, Godel's theorems, etc. I'm not sure if a "higher intelligence" could actually solve that many problems we can't because the problems are intractable.

Not pushing the boundaires of human knowledge, but requiring skills which are not easily learned without experience and within which there is a huge variation in human ability.

Can you really say that the existence of an agent (or a million) matching or exceeding the 100th percentile in many different skills would not be a massive change?

I don't think it changes the human condition.

If any kind of "Terminator" threatens the human race I think it is something that can understand individual psychology and seduce anyone.

Why? Is it arrogant to assume that Turing machines are at the limit of what is possible to compute? Perhaps, but we also have good reasons to believe that to be the case. If you're just talking about things like computational speed differences or being able to control powerful machinery and weaponry, then those are already available for humans to use, and can already be used by humans to do bad things much more effectively. It seems to me that any future improvements in computational speed, memory capacity, etc. will also continue to be available to humans as well as any strong AIs we might develop.

This will invalidate fantastical accelerationist woo until such evidence is presented, and -in the current state of the art research within relevant fields- it's not even certain we can know what would constitute such evidence _in principle_.

If the expectation of a superhuman AI takeover is built on five key assumptions that have no basis in evidence, then this idea is more akin to a religious belief — a myth

My problem is, even this is wrong. I do not find the "five key assumptions" he cites to be the foundation of beliefs that a dangerous "Superhuman AI" could emerge. So to me, this article simply fails right from the jump. The thing he is arguing for and/or against just don't, collectively, constitute a well-developed argument for the overall thesis. Not in my opinion anyway.

To go through them one by one:

Artificial intelligence is already getting smarter than us, at an exponential rate.

I don't know anybody making this claim, nor is it apparent that belief in this is necessary to think that dangerous "Superhuman AI" could be a problem in the future. I guess it comes down to what you think Kelly meant by "soon" in the preceding sentence.

We’ll make AIs into a general purpose intelligence, like our own.

It's the "like our own" part of this that I have a problem with. Yes, I think many people believe that we will create some manner of "general intelligence", but I also believe that a lot of researchers acknowledge that it may not be 100% "human like". But this may be my own bias showing, as I personally harp a lot on the importance of the distinction between "human level" intelligence and "human like" intelligence.

We can make human intelligence in silicon.

This is probably the most sound one of all, but here's the thing... saying "there is no evidence for this" is simply wrong. To hold to this is to completely ignore all progress made building AI in silicon to date. And while we don't yet have "general intelligence", I don't see how one can just ignore incremental progress completely and say "there's nothing there". I would argue that the null hypothesis should probably be that general intelligence on silicon is possible that that the burden of proof is on the people arguing the opposite.

Also, not everybody thinking about / working on AI is focused on the "in silicon" approach. The majority? Yes. But I'm sure there are a handful of people working on ideas related to DNA computing, chemical computing, growing synthetic brain matter, etc.

Intelligence can be expanded without limit.

To me, this may or not not be true, but it's completely irrelevant to the thesis at hand. The question isn't "can machine intelligence be unbounded?", but rather "can machine intelligence exceed human intelligence?"

Once we have exploding superintelligence it can solve most of our problems.

Again, this strikes me as mostly irrelevant to the question at hand.

So yes, this article raises some interesting points, but I don't see it accomplishing what it seems to purport to do.

It could be that there's no way to do that in silica without outstripping plausible bounds on hardware size or power use, I don't know...

I sense that your arguments are in concordance with TFA, broadly speaking. Also, I suspect you read it rather uncharitably for reasons having to do with tone or stated thesis. Either that, or my reading comprehension is failing me. Maybe I want it to be more convincing that it really is, as i already laid bare my sentiments towards accelerationist woo.

I'll take that as a compliment!

All joking aside, I'm not saying every word of the article is wrong or anything. Maybe I'm just being too literal or pedantic or something. It just seems to me that the article sets out to make one very specific claim, and I just don't think it succeeds in making a strong case for that specific claim. Is there some interesting discussion there? Yes. Are there things that Kelly says that I agree with? Yes, absolutely.

So maybe we're not too far apart on this after all.

Agreed.

Dunno why he thinks this: "At the core of the notion of a superhuman intelligence — particularly the view that this intelligence will keep improving itself — is the essential belief that intelligence has an infinite scale."

The notion of self-improvement is both powerful and not assumed to be 'infinite'.

I figure it goes like this:

Q. Is AI potentially dangerous? A. Yes.

Q. Will it be human-like? A. Probably not.

People tend to couch these discussions as a series of anthropomorphisms.