The place where AI alarmists seem to forget this most often is in computational complexity, and particularly in the power of super-hyper-computers to simulate reality. Bostrom in particular doesn't seem to appreciate the constraints complexity theory puts on what an AI could calculate (unless he thinks P=NP).

Here's a response about why complexity theory may not constrain AI that much: https://www.gwern.net/Complexity%20vs%20AI

It is something "AI alarmists" think about.

What's his reason? I got halfway through that and realized I didn't understand.

That even if you see diminishing returns to what you can accomplish with greater intelligence, a small advantage will still compound into a large one over the theoretically infinite lifespan of an AI.

> the theoretically infinite lifespan of an AI

Infinite? Why isn't AI subject to the heat death of the universe?

It just might be smart enough to figure out a way around that.

Naturally , Asimov already wrote about this. _The Last Question_ and _The Last Answer_

Really,Asimov is more on point and more fun to read than all these Superintenlligence pundits

Ooh thanks for this. I've read THQ at least once before, but am really enjoying reading it again. I really liked this bit, emphasis mine as it relates to our current spectating of a dying ecosystem:

"Unhappily, Zee Prime began collecting interstellar hydrogen out of which to build a small star of his own. If the stars must someday die, at least some could yet be built."

He actually even wrote about the current debate. In his multiverse robots were actually banned from earth by bigots at some point.

I miss him I only started to read his book after his death. RIP Good doctor

s/theoretically/practically/, and I doubt it changes the conclusion that much (in practice).

I'll settle for "effectively." Deal?

Deal.

My point being, there's plenty of time before coming heat death starts to cause problems :).

The thing is it doesn't need a huge leap in any specific thing, it needs a quantum leap in anything at all. Complexity only constrains you from solving all problems of a given type in reasonable time, not from solving any instance.

There are even many things that we already suspect are just engineering challenges. Like rocketry, hacking and robotics. Even if an AI can just act like a million engineers and scientists working for a decade per second, without any higher level of thinking, it could do some incredible things.

Why would they be any more constrained than our biological brains?

Humans are able to calculate plenty with a relatively small amount of mass and energy.

I mean that for a lot of things, doubling our brainpower would would only extend our predictive horizon by a tiny amount.

There is a large class of problems for which you have fairly good solutions up front, but where even vast amounts of additional computing power don't help you do much better.

> I mean that for a lot of things, doubling our brainpower would would only extend our predictive horizon by a tiny amount.

Unless you come out with some algorithm that is sub-exponential for the important cases, and only gets slower for stuff you don't care about, or happen very rarely in practice.

People invent those algorithms all the time, for lots of kinds of problems.

The laws of the universe forbid steam locomotives in space, and yet we reached mars regardless. And here we are, barely even regular Inteligences.

The laws of computational complexity seem like a stronger candidate for a hard limit, but that only means that to get smarter they will require more matter to distribute comoutations across.

The battery limit proposal is just silly.

Therefore, to paraphrase Cato, AI research must be destroyed.

That "quantum leap in battery chemistry" is obviously physically possible, given that animals exist...

Or even forgetting biology for a second:

- gasoline is a liquid battery with much greater energy density than our solid ones

- nuclear batteries are a thing, too

> A superintelligent being isn't going to be able to build a superliminal reactionless drive if the laws of the universe say it isn't possible.

It isn't, but if it wants one then it will likely go and discover the actual physical constraints.

> More relevant, a superintelligent being isn't going to be able to enslave us all with robots if the laws of chemistry don't permit a quantum leap in battery chemistry.

That's what differentiates even a somewhat intelligent AI from technologies we have today. Enslaving us with battery-powered robots is but one way to reach the goal. There are many others. If we stick to robots, you have gasoline-powered robots (gasoline is a liquid battery that's still 30x as dense as our solid batteries). Then you have nuclear batteries, which we know are feasible, but don't exist because of political reasons. And then again, robots are but one way to reach a goal...

I think a lot of people forget one thing: intelligence is basically as good as the info you're putting into it. So even if you have a tremendous AI, it can only be as smart as the info it can access. It could only have god-like powers in realms from which it could extract some information. It's still very much subject to being blinded by ignorance (in the most literal sense of the word).

But I do agree that, if it has complete information, an AI could potentially beat us at anything that requires thinking.

Regarding enslavement with robots. Superintelligent means that if it is not feasible to enslave humanity with robots, then SAI will find feasible way: human collaborators, genetically engineered fungi, backing of puppet president candidate, fabricating external threat and convincing people that only it can handle it, and so on and so forth.

All matter is, in principle, a battery. There are no physical laws that prevent near limitless energy storage.

There are if you want to have a functioning computer in the middle of all that energy. There are laws governing heat dispersal (speed of light in only 3 dimensions) and laws governing the collapse of information processing structures into black holes.

Sure, but none of those laws would prevent a superintelligence from generating more than enough electricity to power itself, in principle.

Indeed, this seems like pointing to the speed of light as a limit in a discussion about building faster cars.

AI's powering themselves isn't the objection. Exponential increase in capability is. And that requires you to deal with things like heat dissipation, which are real problems for our current "dumb" computers.

It's not a huge problem for humans. I see no reason to think that a computer say, 10x as powerful as a human would have major problems with heat dissipation.