Is there any reasonable assumption to be made that AI has reached it's zenith? I don't think there is. AI improves over time. That's all it does, and barring some civilizational catastrophe, that's what it will continue to do.

> AI improves over time. That's all it does, and barring some civilizational catastrophe, that's what it will continue to do.

Why? Why might it not stop improving, like lots of things, burning paraffin for heat to pluck one example out of the air?

Because so far it continues to trend upwards? And it's nowhere near the efficiency of the human brain, so we have good reason to suspect we should be able to at least build highly-interconnected AIs with at least the efficiency of the human brain. A couple million of those alone would be unfathomable.

Probably not any time soon, but insofar as we have reason to suspect there's an upper limit (which we don't, IMO), we're not even close to the upper known limit.

> And it's nowhere near the efficiency of the human brain, so we have good reason to suspect we should be able to at least build highly-interconnected AIs with at least the efficiency of the human brain.

Why? I really don't get it. Many, many commenters here and elsewhere have used the existence of the human brain as a supposed proof that we can create AIs with human-level intelligence. I just don't see why the conclusion follows from the observation. Can anyone flesh out the argument in more detail?

Sure, it’s essentially that intelligence appears to be substrate independent. That is, it is not the case that mammalian brain matter is some completely unique material in the universe naturally imbued with “intelligence,” but that it’s a physical structure that does a thing called “intelligence” under the right electrical, thermal, and chemical conditions.

The brain is obviously immensely complex and we’re not close to producing a computer like it any time soon, but there’s no reason to believe we never could in principle. E.g. you could imagine a machine that ”scans” one structure of molecules and “prints” an identical structure of the same compounds. If you were to build this machine and apply it to a brain — copy the exact configuration atom-by-atom — you’d produce an identical intelligence as the initial brain. There’s no known law or fact about the universe (or intelligence) that precludes this. [0]

So if that’s true, and it’s true that there are strong incentives to eventually develop human or superhuman intelligences, then we will one day develop at-least-human intelligences (assuming we don’t get annihilated by a meteor or nuclear holocaust). The worst case scenario is we have to build a facsimile of a brain, which would likely take a very very very long time indeed, but it could be the case that human engineering (as opposed to natural selection) can find a much shorter route to the same destination.

Regardless of which route we end up taking, regardless of how far down that road we are, regardless of whether our current route is a dead-end, the route and the destination do both demonstrably exist: natural selection found one route inside the skulls of some hairless apes.

[0] Note you’d presumably also produce an identical consciousness, but this can of worms need not be opened! That’s a separate topic so we’ll steer clear for now.

I'm sorry, I still don't get it. I agree that in principle there is no reason human level AI couldn't exist, but that doesn't mean that it can happen on planet Earth. Some practical, essential step on the pathway may be impossible, just like in principle there's no reason that life couldn't arise on Pluto, or the proof or refutation of the Riemann hypothesis might be produced next time I generate a random string in Python, but practical, essential steps are impossible.

Do you have something in mind? Both of your examples are referring to spontaneous results out of a random system, and that’s not what any engineering is. If it were, we would have ~0 modern technology at all.

Sure it’s possible that there’s some barrier, but no one can point to what it is or even what it hypothetically could be, so why assume it’s there? Especially given, again, we know for a fact that a random process already achieved it.

Because when we reached the limit of what we could do with paraffin, we moved onto other methods of producing heat. Now we make superheated balls of plasma as hot as the sun using deuterium, tritium, lasers and magnets.

Sure, but your comment suggests that AI will continue improving indefinitely. Why might we not, as we did with paraffin, reach the limit of AI?

Paraffin is a means, not an end. The end goal is to generate heat, and we have all kinds of ways to generate heat now that are superior to paraffin. The future of AI may not depend on diffusion models or even a silicon substrate, but the incentives to improve it are clear, and it will get better.