I think I disagree. Even if it is for some reason impossible to make something that is more intelligent than a human (which seems implausible but I can't rule it out), it still might be possible to make something as intelligent as a human but which runs faster and can improve itself, and that would be enough to create the positive feedback that could lead to a sudden catastrophe.

Discussions like this are interesting because the arguments on one side seem ridiculous, yet logically valid and simple to understand/articulate; the arguments on the other side are complex and difficult to formulate into words.

I just got out of the shower, so I'll add my recent showerthought: In a human brain, each neuron is a CPU in and of itself; this means that every neuron in the human brain can run in parallel. In modern computers, the "perceptrons" are virtual, and there are only between 1 and 4 CPUs to allow activation of a specific "perceptron" at a given time.

So if you have a biological brain with 100 billion neurons sitting next to a virtual brain with 100 billion perceptrons, the virtual brain will run 100 billion times slower even with equivalent intelligence.

I hear you on the parallelism, the flip side is that each neuron doesn’t need anywhere near the flexibility of a general purpose CPU.

I think many of the AI algorithms can run on GPUs, where core counts can get into the thousands (still very far from millions). But these are largely independent — the interconnects between GPUs are not on the scale of neurons.

My guess: we develop specialized hardware that, over time, will increasingly resemble organic brain structure.

"Core counts" on GPUs are quite misleading - what are called "cores" on GPU hardware would be called "execution units/ALU's" on the CPU. GPUs have significantly-enhanced memory throughput, and can use a combination of SIMD (with masked/predicated instructions), barrel processing and genuine multicore to excel at embarrassingly-parallel compute. But they're not magic.

And if what you care about is running a wide variety of mainstream stats or machine learning algorithms, the "specialized hardware" you mention is the modern GPU, or something very much like it - I don't see any low-hanging fruit that might make some other HW architecture more feasible. Fixed-point compute might get there someday, but it's way too fiddly at present - it's not really clear how to "tweak" mainstream learning algorithms so as to dispense with the large dynamic range that floating point compute provides.

I think creating an AI that beats humans in IQ-Tests is way easier than creating one that is smarter than a human.

> violation of the laws of entropy

why? If I try to reconstruct argument I can only come up with reasoning that would show also evolution violates entropy laws.

An AGI, connected to Micro-Electro-Mechanical switches, could learn to let hot air molecules into a box and cold air molecules out of that same box. But such an AGI would function as Maxwell's Demon, and would break the laws of thermodynamics if they didn't consume a certain amount of energy and give off a certain amount of waste heat in the process.

So intelligence = entropy production and energy consumption, and so it cannot grow out of control and transform the planet overnight.

That is only true if you also assume an adiabatic AGI. There is nothing "intelligent" about an optical thermometer linked to a gate.

The reason Maxwell's Demon cannot exist is that such a device would need to produce entropy

Yes, and any intelligent device would also need to produce entropy and release heat.

Which means that the level of intelligence an AGI can obtain is limited to Earth's energy input. And the impact that AGI has on the planet is also limited by Earth's natural processes.

Which means that a hypothetical paperclip maximizer that turns humanity into paperclips and that becomes too intelligent to be turned off is a ridiculous scenario that ignores the potential limitations of an AGI.

That limit is extremely high. Total solar input to the earth is about 10^17 watts. Humans require about 100 watts to produce 1 brainpower -- if AGI is similarly efficient it could produce 10^15 brainpower, or 10,000 brainpower for every human.

If 10,000 people's entire purpose in life was to turn you into paperclips, I don't think you could outrun them for long.

Well humans aren't solar powered; it takes thousands of tons of plant matter and hundreds of tons of animal matter in order to sustain each human. And it takes hundreds of millenia of experimentation with the real world in order to create a world like the one we currently inhabit. If humans harness 100% of the Earth's solar energy to produce a population 10,000 times its current size, it would still be unsustainable. Likewise, an AGI would need to harness that solar energy to produce silicon wafers and to manufacture chips; it wouldn't dedicate 100% of its energy to thinking.

We don't see evidence of life on other planets, and certainly not any evidence of superintelligence gone awry; a planet filled with paperclips is in an extremely low-entropy state. Evolution can happen because human beings, despite being low-entropy beings, turn low-entropy things like bananas and apples into high-entropy things like poop and carbon dioxide. This delicate process is likely why humans are relatively easy to kill, and why a superintelligent AGI would be unlikely to overpower mankind's dominant position on Earth.

So the assumption that a superintelligent computer program could suddenly harness 100% of the Earth's solar energy and outsmart humans to the point where it can't be turned off is still a highly unlikely prossibility. The arguments in favor of AI danger are easy to make but likely wrong, the arguments against AI danger are complex, require lots of thought, but are most likely correct.

To be honest I fully agree that in many senses AI will never be a catastrophic danger to humanity. I just do not see how entropy is part of that.

AGI Co-learning changes the dynamic significantly. If human beings could co-learn, then I think the analogy to AGI would hold a lot better.

“employment” or “usefulness” isn’t the threat.