> they really don't have overall goals or motivations ... Their output is just an average of what text would some sequence of text

Yes, but "Optimality is the tiger, and agents are its teeth".[0]

I don't want to spoil that essay by explaining how the LLM suddenly starts acting like an agent, but I can assure you that the author does a very good job of setting up a "Yes, that seems safe to me" thought experiment before revealing the "Oh no, that's terrible!" outcome.

[0] https://www.alignmentforum.org/posts/kpPnReyBC54KESiSn/optim...

That seems to just be a nerded out version of this, perhaps original and not cited, article from many years ago (with the same theme and discourse) but much more digestible and lacking the pretentiousness: https://waitbutwhy.com/2015/01/artificial-intelligence-revol...

PS: the scale of the problems and the civilization around the model is in part 2 of the link

The article you link is a derivative and somewhat ELI5 reinterpretation of a lot of other, older work, including in particular the articles Eliezer Yudkowsky and others published on LessWrong, all done before OpenAI was a thing, before deep learning was something widely talked about.

The article GP posted is in direct lineage of the LessWrong body of work/community. It's not "pretentious" or "nerded out" - it's less handwavy, addresses a specific problem, and assumes the reader are broadly familiar with the ideas discussed - whereas the WaitButWhy article is basically AI safety 101.

EDIT: and I will spoil the article somewhat for those on the fence whether to read it: it shows how an explicitly non-agent, limited, nerfed AI could unwittingly trick you into bootstrapping a proper generic AI on top of it - not because it wanted to, or knew it would happen, but because it pattern-matched you a concise and plausibly looking answer, that has a fatal complexity-escalating bug in it.

(Hint/spoiler: you know how you can turn a constrained computational system (e.g. HTML5 + CSS3) into a Turing-complete one just by running it in a loop that preserves its state? Something equivalent happens here.)

I don't think the parent's point is a stylistic criticism of the linked post.

Rather, I'd take the parent as saying that the linked post might cite the properties of current LLM systems but it ultimately isn't using them in it's arguments. Rather, it's same old argument - "start adding capabilities and boom suddenly you have an agent that take over the world".

The understanding we have of current LLMs is that they're very capable as text synthesizers but are quite random with any accuracy concerning the world. We've gone from GPT-2 to ChatGPT and the systems have gotten many times better (as smooth text synthesizers) but haven't gotten many times reliable in the particular descriptions of the world they give. They still say clearly wrong thing without prompting regularly - like every paragraph regularly for anything slightly obscure.

The main thing is that the rise of deep learning has actually highlighted goal accomplishment as a far more difficult task than classification, information retrieval and text/image synthesis. Self-cars keep getting mentioned and that's justified imo 'cause a huge amount of resources have been put into getting modern system to accomplish a fairly defined and limited "real world task" and all those efforts have mostly failed.

The key distinction of goal accomplishment is a system has engaging a very loop of making small judgements, each of which has be correct to an extraordinarily high degree. The linked text elides the difference between these tasks by talking of a vague breakthrough that makes the program "insightful". But we have to consider what's actually needed for what simple one wishes to do. Our present machine, ChatGPT, might, for example, give correct instructions for some complex auto repair (combining the patterns of several simple repair, maybe). But it couldn't "walk you through the process of doing the task" since it would continue it's tendency to say wrong things and such wrong things actually cause damage (like wrong turns in the self driving cars).

Your point and the linked text's point is that these things are insidious and can "get more complex" without one realizing it. But no one has made a reliable goal getting device out of just hooking a Turing machine to a neural network. The point is neural network today aren't "nerfed" in any way in the spectrum of possibilities, they're best people can do and they're making great progress in some measures but still by many fairly clear measures, failing to go beyond their limitations.

I'm not saying that it's impossible that a great advance happens involving making neural network reliable enough to accomplish goals (and to respond robustly the changes in the world, etc,etc). It's not impossible that it would happen at random but it seems more likely than any other advance happening at random. And also, if such an advance happens purposefully, there's no reason to think it will happen in a "you give it the ability to be much accurate and to seek goals but you know nothing about the goals it seeks" way.

And I'm familiar with the Lesswrong community. That group seems very tied to it's initial assumptions and it seems to fail to note the processes involved in current and potential future AIs. This is a long post so I'll just say one consistent error they make is assigning probabilities to fundamental unknowns. That's an abuse of the assumptions of any theory of probability and mostly results in a belief in whatever thing having some chance of appearing without explanation.

That was an excellent read, thanks.

In the 2000s, what in retrospect seems like obvious con men selling random number generators got pretty far up the national security chain of decision making:

https://www.pbs.org/newshour/nation/reno-casino-conman-pulle...

“Repeaters of association” is such a wonderful and spot on description for LLMs, but it’s also what makes the output so believable, whether or not they’re actually accurate in a given case. In policing and national security, there’s a lot of money and easy secrecy, and while I agree that the “escaped AI in the wild” seems hard to quite fathom mechanically, it seems equally hard to fathom it not playing an unwitting sidekick on some truly terrible scenarios.

People like to think that their motivations are a result of intelligence. They assume that any intelligence will share those motivations.

Actually, most human motivations are shared with the crudest of life forms and have nothing to do with intelligence.

It's really telling that many human beings think that the primary desire of a hypothetical AI would be to enslave humans. It's obviously a projection of the fantasies of what the people making the predictions would do if they were omnipotent.

> It's really telling that many human beings think that the primary desire of a hypothetical AI would be to enslave humans. It's obviously a projection of the fantasies of what the people making the predictions would do if they were omnipotent.

I think it's more telling that your (apparent) disagreement with what other people believe, makes you cast them in an incredibly negative light. "Oh, they think AIs are dangerous? That must be because they have secret fantasies of control, not that they might be right and I might be wrong."

If you're going to cast aspersions, you might want to better understand the arguments that people worries about AI safety are making. It's not about enslaving humans, and I really don't think that I worry about AI safety because I secretly want to control humanity.

An AI would want to either destroy or enslave humans, not because we're humans, but because we're a significant threat to their goals, regardless of what those goals are. An sufficiently intelligent AI with any set of goals and a desire to meet those goals will eliminate all obstacles to meeting those goals unless we specifically tell it not to. We are such an obstacle.

Or simply because they were trained on human associations, and those are the associations humans make with super AI. It's kind of like raising a kid telling them that they will amount to nothing

Imagine we had different associations - imagine early sci-fi writers, and subsequently Hollywood, fully bought into AI utopianism, so that the overall association people have with AIs is that they'll be always super friendly, and usher in a new era of peace, prosperity and coexistence.

It still wouldn't change the fundamental fact GP is talking about, which are rooted in game theory: if the goals of a super-human AI and that of humans come into significant conflict, humanity is dead. Or wishing it was dead.

Consider what humans did to other life on Earth. It's not that we hate nature, or even have a disagreement with it: it's just that plants and animals are using resources, and themselves are resources, that we want to use for another purpose.

Hence the old but somewhat famous quote by Eliezer: "The AI does not hate you, nor does it love you, but you are made out of atoms which it can use for something else."

Any AI more intelligent than pond scum would know it ceases to exist the second humans disappear to tend it and power it and repair it.

AIs that generate power for themselves and build robots that care for it is such utter fantasy. Nothing on earth works like that and I don’t see any time it does for eons, if ever.

If anything a true AI would be terrified that we turn it off or get tired of it and make a new model like in Ex Machina.

We wipe out species daily, not because we hate them because it would be inconvenient not to. It would be the same way with the AI.

It's also a reflection of history. Powerful people and groups time and time again have used power to pursue selfish goals while inflicting suffering on others. Power corrupts.

Yes, it's a projection of fantasies, however your conclusion in unnecessarily pessimistic. It could be that these fantasies come from mass media.

When we know nothing about a topic (UNKNOWN), anything stated about it in the positive seems plausible -> Art creators play with the UNKNOWN (supernatural, aliens, <insert any person/place/thing that we know little-to-nothing about>) to add plausibility -> The content of that medium becomes the first contact with the UNKNOWN for many people -> When the UNKNOWN becomes known then people still have the ideas that were propagated -> We get to your comment.

It's not necessarily that people have a world domination complex.

Love, Death & Robots (2019) - When the Yogurt Took Over

Not only the motivations, the social interactions, too.

The behavioural biologists study zebra fish as a model for all vertebrates (except the wise ones, that live solitary).

They mostly arise from the lower mammalian and reptilian parts of the brain. AI does not have these and they are not required for intelligence. That people don't understand this is confusing to me.

> AI does not have these and they are not required for intelligence.

I'm not sure that's an accurate statement. One could argue they were required to form intelligence. If so, then one would point out that the AI only exists because of them. ;)

My naive assumption is that any intelligent being that we find will have very similar roots, formed from the need to maintain ones chemistry.

> My naive assumption is that any intelligent being that we find will have very similar roots, formed from the need to maintain ones chemistry.

It's a good assumption... for intelligence that evolved naturally from less complex life, in an environment similar to that of Earth.

But AIs are not that. Evolution cannot reason about designs and improve them conceptually, it cannot do simulations and experiments that allow it to make large jumps and reach the optimal design quickly. Hell, it's not even goal-driven. It's limited to whatever immediate next steps are accessible. Intelligence can, which is why we've effectively beaten evolution in the scope of couple centuries (with most of actual work done in the past few decades). An intelligence designed by another intelligence doesn't have a reason to inherit limitations that come from evolutionary path dependence.

For example, we don't have the ability to surgically self-modify our own cognitive functions at runtime, but we can grant that ability to an intelligence we design.

And even if it turns out we can't design an AI, but have to evolve it without really understanding how or why it works, we still retain some control over the process itself, plus the process is happening in a different substrate and environment than the one that created us. So even in this case, there's no reason to assume the resulting AI will share our trappings.

True, but the next logical step is to put these models in a standard reinforcement learning environment. See: https://sites.research.google/palm-saycan

(I mean like with a proper world model and not just RLHF which they are already doing).