I find Carmack incredibly arrogant to assume the he alone will be able to solve AGI. I mean the guy is a good programmer, no question about it, but let's get real: his major credentials come from having built 3d game engines, which by today's standards are quite primitive. His work at Facebook / Meta is meh at the most, the Quest didn't revolutionize VR by any means. Becoming rich from the sale of Id to Zenimax is not proof of genius. I admire his achievements, but I think he's way overrated.

> his major credentials come from having built 3d game engines, which by today's standards are quite primitive

Primitive now but in their time they were groundbreaking and incredibly difficult to make work on the primitive hardware of the time and he kept pushing the envelope for a long time. Carmack is a bit of a genius, but that doesn’t mean he’s right about AGI. My own take is that actual General Intelligence may rely on any physical phenomenon from the macro to quantum effects including aspects of reality we don’t know about yet. It’s really hard to simulate things when you don’t know they’re involved. We won’t know until we try so I guess we’ll see what happens.

The initial breakthrough that eventually leads to the creation of AGI will come from one person. The creation of AGI will likely be a incremental process involving small improvements on that breakthrough, made by many people.

This is generally how science works. 99% evolution in small incremental advancements, 1% revolution in a stroke of genius that opens up new areas of science.

It's pretty easy to build AGI if you relax Asimov's laws of robotics. If you make machines that are allowed to harm people they can effectively build up all the intelligence they need by the following algorithm "mimic others and don't die". Isn't that more or less what humans do? It seems odd we'd expect a general intelligence to come out of something that didn't itself have a general existence. That general existence includes making mistakes in order to learn from experience and making mistakes sometimes entails accidentally harming others. Sometimes it involves coming to an erroneous conclusion that causes that intelligence to deliberately harm others which then gets reprimanded and other agents learn from that.

nothing about asimov's laws have anything to do with intelligence, or even robotics. they came from some strange paranoid "robots harming humans obsession. They aren't even remotely implementable in any possible way.

Anything intelligent (meaning universally effective) is harmful unless perfectly aligned with humans (hard to get).

Even humans (being barely intelligent, weak) are harmful to other humans (look at elites around the world: wars, coups, assassinatios, propaganda, etc).

Unless there is a fundamental barrier that handicaps intelligence at the human level (highly unlikely), AGI will eventually leave people far behind and destroy us in the process (like we destroy an ant hill, to build whatever we need for reasons incomprehensible to ants).

I guess robots that can kill people on their own "will" would be perceived by humans as having chaotic behavior , and wouldn't be of much use to us. So why build it?

Anyway I think robots or other devices can't have will either way. For example if someone plants land mine and years later this land mine kills someone. It is not land mine's will. But of person who planted it. You can't sue land mine.

That John Smith left his paperclip maximizer running will be cold comfort while we're all getting reconfigured into paperclips by a particularly deviously formulated by AI bacterium.

Sometimes, there are good reasons to just not build a thing. Takes a while for the idea to become apparent, but once you start thinking in terms of actually being responsible for second order and higher outcomes, it becomes much easier.

But it's cultural dogma that Businesspeople aren't responsible for any externalities related to their endeavours. I don't see how that's going to change anytime soon. Move fast and break things!

Take heart, though; the paperclip maximizer will probably just "want" us to watch ads and buy shit. A slow death, via titrated amounts of The Entertainment from Infinite Jest. At least it won't hurt too much.

You need to insert a limited lifespan alongside "don't die"... and maybe also reproduce as much as possible

Of-course it will be "simple". Human brains are "simple" too. Edit: To clarify, what I mean is it will appear to be simple once we have understood how it works and how to implement it ourselves.

But the path towards this goal is going to be anything but simple. To begin with it will need a multiple order of magnitude improvement in computational power. I primarily view this as a hardware problem.

I had this view once, and then I married a microbiologist.

Nothing in biology is simple. Nothing. Even the simplest things are orders of magnitude more complex than I ever could have imagined from my lofty platonic realm of numbers and functions.

Life is fractal. Every time you zoom in, there is just as much complexity as the previous zoom level.

Except it’s all basically clever tricks to cause gradient diffusion to do what is needed.

Absolutely, but I suspect much of this complexity is accidental / incidental in nature. It is needed to support the physical aspects of the biological processes. The essential (algorithmic / mathematical) part will be quite simple and can be emulated in a few thousand lines of code.

Complex algorithms are not energy efficient and since the brain is highly energy efficient we must preclude such algorithms from consideration.

> I suspect much of this complexity is accidental / incidental in nature

From what little I know, I am unwilling to make that assumption, but I do think that starting with a relatively simple algorithmic/mathematical biology-inspired take and slowly ratcheting up the complexity as needed is a reasonable way to proceed.

Agreed!

Yes, everything is simple if you ignore all the details and most of the general concepts.

Some details are more important than others and some irrelevant.

I posit the relevant details will be equivalent to a few tens of thousand lines of code since the brain is a highly (energy) efficient organ and there is no room for complex algorithms.

Make sense. A fun argument for why brain's underlying "algorithm" should be simple is that evolution didn't take that long to go from creatures with primitive cognitive functions to thinking-talking humans. The sensible solution, just like the way evolution solved many other problems, is to replicate what's working in a massive scale, similar to how we build artificial neural networks. This argument was popularized by jeff hawkins in his thousand brains theory, I believe.

Punctuated evolution doesn't mean that evolution was fast, at least as I see it: instead evolution has solved problems in the past and created latent capabilities that can be rapidly engaged through small mutations or epigenetic changes like hormonal changes that get passed on generationally through early development.

That is to say that while humans developed higher intelligence over a relatively short time, that was created from a novel combination of evolutionary abilities rather than a novel evolutionary invention.

That’s because AGI is all about taking investor money and making it your own, which really doesn’t require complex code.

GPT3 prompt:

Following JAX code is the essence of artificial general intelligence. It uses all simple tricks and techniques in tandem to achieve super-human meta-learning abilities.