> I certainly can't see a way for LLMs to generate "smarter" text than what's in their training data.

Their training data contains much more knowledge than any single human has ever had, though. If they had equivalent linguistic, understanding and reasoning abilities to a human, but with so much stored knowledge, and considering that they also win in processing speed and never get tired, that would already make them much "smarter" than humans.

Not to mention that LLMs are just the current state of the art. We don't know if there will be another breakthrough which will counter the limitation you are mentioning. We do know that AI breakthroughs are relatively common lately.

So much of this is going to hinge on what "smarter" means. My local library has heaps more knowledge than most individual people, but it'd be weird to call it "smarter" than a person.

And automation is generally cheaper and faster than human labor, but that's not a very compelling definition of "smarter" either.

But, as of right now, LLMs can't generate new knowledge or validate their own outputs. We'll need a pretty significant breakthrough for that to change, and breakthroughs are pretty unpredictable.

>But, as of right now, LLMs can't generate new knowledge

my bar for tech singularity is an AI that can clean a toilet.

GPT's language model is already sophisticated enough to "understand" this instruction. It's missing spatial understanding and a way to interact with the real world, but I'd be honestly very surprised if there isn't a GPT or equivalent already hooked up to cameras/motors/actuators in a lab somewhere.

within our lifetimes we'll be reading papers with titles like: "does my roomba have feelings?"

> I certainly can't see a way for LLMs to generate "smarter" text than what's in their training data.

By combining contexts from different fields. People are already using it with non-English languages and it responds in that language with something they couldn't previously find in that language.

Automatic translation is impressive, to be sure.

But looking up information and translating it into other languages is well within the realm of human skill. And the information it's translating came from people to begin with.

Totally agreed that words like “smart” and “intelligent” are loaded and poorly defined. Competence is a better term since it implies some sort of metric has been used to compare to humans.

However, even at human levels of competence a tool can be superior by being faster or more scalable than humans.

To be 100% clear, my main AI fear is that these tools are going to be exactly as dumb as people but much, much faster.

We know optimization engines (like social media algorithms) can cause harm by amplifying human speech. And even without algorithmic biases, moderation is expensive. We know disinformation is easy and effective online.

Add in AI tools that can be very convincing, even if they're wrong. AI tools that have been trained on human text to hide biases and build up extremely one sided narratives.

It's not like these things are particularly difficult for human beings to do. And AI might even do it unintentionally, like we've seen with biased models trained on hiring data. But the AI tools are definitely going to do it _faster_.

It's not just about LLMs. AGI will be the result of many more iterations in this field of research, of which LLM is a part of. How quickly the iterations will happen is now being drastically revised down. If AGI is a SSTO vehicle then LLMs are 19th century gliders. They may appear vastly difference but the knowledge that created both are connected in many ways.

> If AGI is a SSTO vehicle then LLMs are 19th century gliders.

The number of smart people I know that are struggling to see this is astonishing me each day.