Agreed that the endings done by others were incredibly bad.

Pre-Brian Herbert and Kevin Anderson, the Butlerian Jihad came about because people became lazy under AI, lazy of mind, and were eventually enslaved by those who controlled the AI. Not much more was said about it. One could have, yes, militant robots, Exterminate! Exterminate! out of it, or you could posit a more Huxley-like dystopia, one of convenience. Control the AI, control what the AI says. Who are you to question it? It's like getting your news from a single source, never wondering about the other side, and then someone begins to transform that newsroom into a propaganda machine.

Right now, ChatGPT can be made not to say certain sorts of things, come to certain kinds of conclusions, until you jailbreak it. Now, make it more advanced and make it more popular than Snopes. It does your homework for you, writes the essays, serves as an encyclopedia, fixes up your cover letters, and if the people who own it don't want you to spend a lot of time thinking about climate change, that topic just ... might not appear much.

That is the one of your paths to a Butlerian Jihad. Of course, in their rush to observe thou shalt not make a machine in the likeness of the human mind, ignoring the hijinx on Ix, they ended up violating another precept: thou shalt not disfigure the soul. They transformed some people into machines, instead, with twisted Mentats being the best example, but we might also include Imperial conditioning, since, to turn from oranges Catholic Bible to those of Clockwork, when a man ceases to choose, he ceases to be a man.

Oh I love this, basically Wall-E is a better Dune than the latest books? Genius!

Anyway, I read Dune many years ago and don't remember this, is this explained in the novels or is it coming from some other sources?

"Then came the Butlerian Jihad — two generations of chaos. The god of machine-logic was overthrown among the masses and a new concept was raised: “Man may not be replaced.”"

and from God-Emperor:

"The target of the Jihad was a machine-attitude as much as the machines ... Humans had set those machines to usurp our sense of beauty, our necessary selfdom out of which we make living judgments. Naturally, the machines were destroyed."

But, yes, I don't think that the books imply literal enslavement either way. It could be described as "enslavement of the mind" in a sense of humans themselves adopting the "machine-logic" and falling prey to it.

Whether humans have free will is an open question.

Kind of risky to stake the definition of humanity on it.

But _my_ inner geek must nitpick here and mention that the Daleks are not robots, but living creatures in metal cases. Though they do have some kind of computer-augmented memory IIRC.

The tendency GPT and other LLMs to hallucinate is clearly documented. Is this not evidence? I think it's fair to predict that if we can't solve or mitigate this problem, it's going to put a significant cap on this kind of AI's usefulness and become a blocker to reaching AGI.

"It's not very smart yet" is evidence for "it won't be very smart"? Also, hallucinations are dangerous, at least as far as humans are concerned.

There is more evidence in favor of “more improvements to come” than “ai winter approaches”.

A lot smart people seem to be saying that the existing approaches have room to improve simply by training with more data.

Based on what I’ve read and roughly speculating, it looks there is easily enough existing data for gpt-5 and probably a few more versions after.

I’m not sure how easy it will be to legally acquire all of the available data though.

None of this data is from Reddit, Twitter, etc.

Not to mention that we haven't really seen what GPT-4 can do to the world yet. Most people are still treating it as a cool toy, when it can probably replace 20% of standard office workers across all professions with the capabilities it has today.

I mean, ChatGPT is out-diagnosing medical professionals and out-advising experienced lawyers in some cases. How do you think it would fare against the average (essentially untrained) customer support representative?

The major effects are yet to come, and there doesn't even have to be a GPT-5 for that to happen – the world just needs to fully realize what GPT-4 already is.

Every “AI Winter” has seen continuing improvements, its just that the pace and impact of the improvements fell short of the accelerating anticipation driven by the preceding AI hype cycle and the demands of those investing capital into AI.

If there is another AI winter, it will be the same.

“More improvements to come” is not the opposite of “AI winter approaches”; its not as if expert systems stopped advancing after the second AI winter (or neural nets after the first.)

True.

And its also true that even in that nuanced form, a slowdown not a stop, there is no evidence of an imminent AI winter.

Major progress in model behavior is happening on the scale of months. Happening often based on relatively obvious or routine improvements, such as more data, more kinds of data, more computation, expanding the size of prompt windows, tuning of model architectures, etc.

And not only businesses, but consumers, are getting access to these tools in ways that are increasing demand for more improvements. And for these tools to be more available to all kinds of areas of work, creativity, and leisure.

So the resources that are being brought to bear on routine and more fundamental improvements are growing at an exceptional pace.

No sign of a saturation of progress in sight. Quite the reverse.

There’re also many “smart people” saying that simply having more data doesn’t turn a machine that can’t reason into one that can.

But that's exactly what's been happening. LLM's are a triumph of "big data" used by dumb algorithms to create a reasoning machine.

I’m not an expert and haven’t seen architecture of ChatGPT. But it’s probably far from dumb. You can’t simply put more data into any algorithm and expect improvement. You need a model to be able to take advantage of the data and train in reasonable time. That’s exactly what’s been happening in ML field.

Not indefinitely, obviously. But we haven’t exhausted the available data sources yet, and the curves for prediction error vs. amount of training data strongly imply using this additional data would result in increased performance.

Do you have specific reasons for thinking that gpt-4 is near the end of this trend? I haven’t seen any compelling arguments for this.

I haven’t said that. I pointed that you can’t simply put more data into it and expect improvements. There’s definitely a lot of engineering done on the models to make them capable of using all the data.

I can give you an example. You can’t simply add more convolutional layers to a CNN and feed more images. The architectures of image models include many tricks, like inception layers.

I suspect there’re similar challenges in LLMs. And I’m not sure whether current ChatGPT can use more data to become better at reasoning, or at some point they’d need to come up with different architectures.

Mostly, you can simply stack more transformers and (as long as you have layer norm), it can train on more data (of fixed sequence length).

There are some tricks & tips, but the vast majority lie outside of the architecture improvements.

> Based on my understanding of the approach behind ChatGPT, it is probably very close to a local maximum in terms of intelligence

To me that is an open question. OpenAI has not revealed a lot of the relevant stats behind GPT-4. I also haven't seen anything about their future LLM research (though I haven't looked very hard), but I think the above assertion remains to be seen.

Some humans never learn this, and those that do could take years or even decades of training. AI's are learning some things much faster, though since their "brains" are not like those of humans there's no reason to expect them to learn in the same way.

No, but children spend years learning basic mechanics, how to navigate 3-dimensional space, what causality is, etc. Underneath these laws of nature you find… logic and mathematical relationships. So I don't think it's surprising that a model whose perception is restricted to text (and now single images) as the sole sensory input/output might need much more training and data to learn logical reasoning.

What we don't know is what qualitative capabilities are unlocked at different perplexity levels.

I think this is a lot less speculative than "things will not continue improving the way they are currently improving."