This is great insight. Science fiction does a great job inspiring technology research, but we forget that real-world accomplishments can be much humbler and still astonishing all the same.

I'm a bit tired of the hype surrounding LLMs, but all the same for very mundane and humbler tasks that require some intelligence modern LLMs manage to surprise me on a daily basis.

But it rarely accomplishes more than what a small collection of humans with some level of expertise can achieve, when asked.

Or accomplishments can genuinely be astounding.

Surely, the LLM models we have today are astounding by any measure, relative to just a few years ago.

But pronouncements of how this will lead to utopia, without introducing a major revision of economic arrangements, are completely, and surely intentionally/conveniently (Sam isn't an idiot) misleading.

Is OpenAI creating a class of stock so everyone can share in their gains? If not, then AGI owned by OpenAI will make OpenAI shareholders rich, very much to the degree its AGI eliminates human jobs for itself and other corporations.

How does that, as an economic situation, result in the general population being able to do anything beyond be a customer, assuming they can still make money in some way not taken over by AGI?

Utopia needs an actual plan. Not a concept of a plan.

The latter just keeps people snowed and calm before an historic level rug pull.

For an intelligence to be "General" there would have to be no type of intelligence that it did not have access to (even if its capabilities in that domain were limited). The idea that that's what humans have strikes me as the same kind of thinking that led us to believe that earth was in the center of the universe. Surely there are ways of thinking that we have no concept of.

General intelligence would be like an impulsive 12 year old boy who could see 6 spatial dimensions and regarded us as cartoons for only sticking to 3.

Humans can survive in space and on the moon because our intelligence is robust to environments we never encountered (or evolved to encounter). That's "all" general intelligence is meant to refer to. General just means robust to the unknown.

I've seen some use "super" (as in superhuman) intelligence lately to describe what you're getting at.

Super feels to me like it's a difference in degree rather than in kind. Something with intelligences {A, b, c} might be super intelligent compared with {a, b, c}. i.e. more intelligent in domain A.

But if one has {a, b, c} and the other has {b, c, d} neither is more or less intelligent than the other, they just have different capabilities. "Super" is a bit too one-dimensional for the job.

I don't think using set notation is really helping your case here but (I think?) I agree.

No. AGI is talking about the "general" human intelligence rather than specialism. An AGI would be as good as a human at both composing poetry and playing chess, both identifying bird calls and proving mathematical theorems. We don't know what a categorically superior intelligence would be so we can't rate some machine on that basis.

The Lem story "Golem XIV" concerns a machine which claims it possesses categorically superior intelligence, and further that another machine humans have built (which runs but seems unwilling to communicate with them at all) is even more intelligent still.

Golem tries to explain using analogies, but it's apparent that it finds communicating with humans frustrating, the way it might be frustrating to try to explain things to a golden retriever. Lem wrote elsewhere that the single commonality between Golem's intelligence and our own is curiosity, unlike Annie, Golem is curious about the humans which is why it's bothering to communicate with them.

Humans (of course) plot to destroy both machines. Annie eliminates the conspirators, betraying a hitherto un-imagined capability to act at great distance, and the story remarks that it seems she does so the way a human would swat a buzzing insect. She doesn't fear the humans, but they're annoying so she destroyed them without a thought.

All it would want to do is talk about Minecraft and the funny memes they saw.

If it's stuck at 12 year old level intelligence then it's not generally intelligent. 12 years can learn to think like 13 year olds and so on.

A set of weights siting on your hard disk don’t evolve to get smarter on their own. AGI does however require the ability to learn new skills in new domains; if you kept exposing it I don’t see why it wouldn’t be similar. But fundamental capacities in how good it is a reasoning / planning don’t have to exceed any given human to be considered AGI.

A 12 year old that doesn’t sleep, eat, or forget, learns anything within seconds, can replicate and work together with fast as light communication and no data loss when doing so. Arrives out of the womb knowing everything the internet knows. Can work on one problem until it gets it right. Doesn’t get stale or old. Only had emotion if it’s useful. Can do math and simulate. Doesn’t get bored. Can theoretically improve its own design and run experiments with aforementioned focus, simulation ability, etc.

What could you do at 12, with half of these advantages? Choose any of them, then give yourself infinite time to use them.

A 12 year old that doesn't interact with the world.

Many believe that AGI will happen in robots, and not in online services, simply because interacting with the environment might be a prerequisite for developing consciousness.

You mentioned boredom, which is interesting, as boredom may also be a trait of intelligence. An interesting question is if it will want to live at all. Humans have all these pleasure sensors and programming for staying alive and reproducing. The unburdened AGI in your description might not have good reasons to live. Marvin, the depressed robot, might become real.

I’m not sure, but it’s possible Stephen Hawking would have been fine with becoming digital, assuming he could keep all the traits he valued. He had a pretty low data communication rate, interacted digitally and did more than most humans can. Give him access to anything digital at an high speed and he’d have had a field day. If he could stay off Twitter.

>simply because interacting with the environment might be a prerequisite for developing consciousness.

We can't even define what consciousness is yet, let alone whats required to develop it.

We may not be able to fully define Consciousness, but we can observe it in humans and develop falsifiable hypotheses, of which there are several.

Take a look at this paper: https://osf.io/preprints/osf/mtgn7

Why do we assume an AGI would not be forgetful or would never get bored?

Because we can change the hardware or software to be what we want. It can write to storage and read it, like we can but faster.

A 12 year old could determine that "AI" is boring and counterproductive for humanity and switch off a computer or data center. Greta Thunberg did similar for the climate, perhaps we need a new child saint who fights "AI".

And then with advancement, reaches teenage maturity and tells everyone to fuck off.

As someone who has teenagers right now, I can confirm that this is accurate.

> AGI doesn’t mean smarter than the best humans.

Technically no, but practically...

12 year old limitations are: A. gets tired, needs sleep B. I/O limited by muscles

Probably there are more, but if 12 year old could talk directly to electric circuits and would not need sleep or even a break, then that 12 year old would be leaps and bounds above the best human in his field of interest.

(Well motivation to finish the task is needed though)

What stops it getting more intelegent? That's litterally its primary aim. Its only limit as to how quickly it does that is hardware capacity, and I'd be shocked if it didn't somehow theorise to itself that it can and should expand its knowledge in any way possible.

Why would the only limit be hardware capacity? Can’t there be an innate limit due to the model size/architecture? Maybe LLMs can’t be superintelligent because they are missing a critical ability which can’t be overcome with any amount of training? It’s not obvious to me that it’s going to get smarter infinitely.

The limit could also be in the training data. If we could train LLMs with a corpus generated by some super intelligent entity, maybe we'd consider them super intelligent even with the same model size/architecture we have today.