It would be a human guiding the ai to guide other humans to end the world.

Not necessarily. If you haven't experienced an AI doing something you not only didn't want, but had expressly asked it not to do, you haven't used AI much.

That's a human problem, not an AI problem.

well it’s a human problem enabled by advanced AI.

Yes, basically a vastly more difficult and dangerous version of not letting young children play with matches, which will hopefully teach us many object lessons about safety and security long before it acquires launch codes or the means to use them.

[delayed]

I think intuitively, intelligence is in some form about being able to predict the reaction that follows an action.

The extreme of intelligence is being able to create perfect statistical distributions for the effect of an intervention in the world. If you had that ability, you could turn it on the question of "what would this human choose to do if I were to give them instruction X" for any X.

I don't present any support for or evidence of:

- LLMs gaining intelligence by whatever means

- Extreme intelligence being possible

- Extreme intelligence being feasible

But I don't think I really need to provide support for the idea that something superintelligent would also be able to predict the actions of people under pressure in the same way someone of moderate intelligence can predict the actions of fluids under pressure.

You seem to describe humanity and AI's ability to predict and manipulate humanity a bit like how a physicist describes the ability of an apparatus to affect a given dynamical system. So let me put it like this: (1) You cannot model the effect f_predict(X) of your probe X with arbitrarily low |f_actual(X) - f_predict(X)|, and (2) you cannot guarantee the impact |f_actual(X)| is arbitrarily large (or that f_actual(X) saturates its codomain) when X in our case is restricted to inputs of the form, "tokens chosen in an LLM conversation."

That's a very sophisticated way to say "words can't convince people to do things".

I'm not sure that's a strong position to adopt in our world, but following your example I also won't try to persuade you otherwise.

That's not what I said, and it's the least sophisticated way I could have chosen to phrase it to engage your argument in a precise way. The closest thing to your summary that I would accept is, "words can't convince people to do arbitrarily crazy things, in a way that you can predict."

> If you had [superintelligence], you could turn it on the question of "what would this human choose to do if I were to give them instruction X" for any X ... But I don't think I really need to provide support for the idea that something superintelligent would also be able to predict the actions of people under pressure in the same way someone of moderate intelligence can predict the actions of fluids under pressure.

So can humans predict these things to a degree. But even having perfect prediction (which I contest with (1)) does not entail perfect control, just by continually computing the effect of, and refining, your X.

> (1) You cannot model the effect f_predict(X) of your probe X with arbitrarily low |f_actual(X) - f_predict(X)|

"You can't know what the impact of your action will be if it doesn't change much"

> (2) you cannot guarantee the impact |f_actual(X)| is arbitrarily large

"You can't necessarily change much"

> (or that f_actual(X) saturates its codomain)

"People can't do arbitrary things". I agree with this statement, but... people can build atomic bombs by themselves? See The Atomic Boy Scout. I don't think it's very relevant to the discussion that you can't make someone phase through a wall.

> when X in our case is restricted to inputs of the form, "tokens chosen in an LLM conversation."

"Yes, I meant all the above specifically in the context of talking to a person, so I can't change my point of view later to be about something other than the persuasive power of words".

> "You can't know what the impact of your action will be if it doesn't change much"

Incorrect translation. No "if" clause need apply.

> "People can't do arbitrary things"

Also incorrect. f_actual(X) is a product of both "how much people can do" and "how much you can control/persuade them." I would model each of these with their own precise terms and relate them back to f_actual(X), and we could continue like this indefinitely, but from now on I'll just give you my reply in English: Not only does The Atomic Boy Scout not prove that an arbitrary person can bring about mass destruction (collecting random radiological material and contaminating the local area is not developing a nuke, nor is it even an effective dirty bomb, because it was neither very dirty nor a bomb; the safeguards were not lack of knowledge, either), but it still remains to be demonstrated that individuals will be radicalized by AI just by virtue of the AI being, like, super-duper smart.

> "Yes, I meant all the above specifically in the context of talking to a person, so I can't change my point of view later to be about something other than the persuasive power of words".

What domain of input do you propose, then? AIs as rational agents presently have no agency but for what tokens they generate. What we hook them up to, and what they could plausibly do with what we hook them up to, has been left unsaid.

Denying the premise, which is that an intelligent being is one who can predict the outcome of actions with sufficient precision to influence the future, makes debate pointless.

I literally granted you that premise:

> even having perfect prediction (which I contest with (1)) does not entail perfect control, just by continually computing the effect of, and refining, your X.

You don't need perfect prediction, you need good-enough influence on enough of the population. Which is something that adtech, social and trad media, and troll farms have perfected already.

We literally have people running around convinced the earth is flat. And a president who is both senile and delusional with access to the nuclear codes.

Hostile ASI wouldn't look like a rogue individual. It would look like a faction, group, or cult which gradually infiltrated and took over political power.

Or possibly inserted itself into critical elements of the tech industry. A digital adversary could do huge damage just by taking AWS offline.

That's just one possible attack vector among many others, most of which aren't well-defended. Especially if the AI itself is being used to harden their defences.

By "hostile ASI," are you speaking in terms of an AI that decides to go rogue independent of human instructions? Or of AI being used by bad actors?

The latter is a more defensible concern. The former is a scifi premise that had more credibility before the current era than since, IMO.

> Especially if the AI itself is being used to harden their defences.

Your use of "is" rather than "isn't" only makes sense to me if the former interpretation is intended. To the extent that you mean to say the latter is also part of it, it's undercut by the defense being AI-augmented. (Unless you mean the bad actors will be the ones in control of all AI, i.e. OpenAI and Anthropic and friends?)

> A digital adversary could do huge damage just by taking AWS offline. That's just one possible attack vector among many others, most of which aren't well-defended.

I think you picked AWS because it's one of the few high-consequence digital systems a rogue ASI could plausibly take down on its own. Anything that could cause actual existential risk would involve more than disruption, but long-term orchestration of systems that are air-gapped, time-gapped, and meat-gapped.

Note: This is a very good argument to never remove the air, time, and meat gaps from those systems. If they do get removed, it will not be the genius of AI, but the stupidity and malignity of mankind that dooms us.

>By "hostile ASI," are you speaking in terms of an AI that decides to go rogue independent of human instructions? Or of AI being used by bad actors? The latter is a more defensible concern. The former is a scifi premise that had more credibility before the current era than since, IMO.

You seem to be moving the goalposts a little, but I don't see why "rogue AI" is less credible after the LLMs. It's obviously more. And we're just at the start of constantly running agent style harnesses.

>To the extent that you mean to say the latter is also part of it, it's undercut by the defense being AI-augmented

And yet, in many cases, those on the offense have a big advantage that any defense lacks. Especially if there's no concern for one's own life, which whether it's an AI or a human AI user removed from the field via his use of AI to delegate doesn't need to have (or they're just mad/fanatics and don't have in general).

>Anything that could cause actual existential risk would involve more than disruption, but long-term orchestration of systems that are air-gapped, time-gapped, and meat-gapped.

As if Stuxnet didn't exist?

> Hostile ASI wouldn't look like a rogue individual. It would look like a faction, group, or cult which gradually infiltrated and took over political power.

Or like a massive AI datacenter boom of questionable long-term profitability? Maybe the end goal isn’t profit but large amounts of compute?

>So can humans predict these things to a degree. But even having perfect prediction (which I contest with (1)) does not entail perfect control, just by continually computing the effect of, and refining, your X.

And why do you need perfect prediction and/or perfect control? "Good enough for the results you want" prediction and control sounds perfectly fine.

We have had cult leaders that had neither perfect prediction or control, and still convicted hundreds of thoysands of people to live as they say, and a decent chunk to even suicide on command!

> What makes you think that persuasion isn't capped at some lower level than world-ending?

People can be persuaded to kill themselves (see koolaid). People can be persuaded that arbitrary others are evil and must be exterminated (see Hitler). People can be convinced that dying for the cause is worth it (see 9/11 or kamikaze). People can believe that bringing about the end of the world would cause them to be saved into heaven (see Christian Zionism).

Why do you think it would be hard to convince any of these people to do something really stupid?

In fact, people don't even need to want to cause destruction to do something dumb - "I'm trying to save the world with free bio-fuel, help me create this bacteria" ("and I'm very smart, smarter than Elon Musk") - boom, pandemic.

It takes only one person to fall for it.

> People can be persuaded that arbitrary others are evil and must be exterminated (see Hitler)

If you are suggesting Hitler was persuaded that the Jews were evil, I don’t think that’s correct, he grew up in an environment where many people did believe that but they did not need to be persuaded, just like they did not have to be persuaded into their religion, culture etc. They just accepted the idea without much thought, which does not equate persuasion.

If you are saying the Allies were persuaded that Hitler was evil and must die, I think that may be closer to the truth since they knew the cost of invading Germany would be astronomical, potentially costing their own destruction, and they still stopped only when Hitler was taken, dead or alive. Some serious persuasion was needed for that.

> They just accepted the idea without much thought, which does not equate persuasion.

The first half of your sentence sounds like really swift and effective persuasion, to me.

>If you are suggesting Hitler was persuaded that the Jews were evil, I don’t think that’s correct, he grew up in an environment where many people did believe that but they did not need to be persuaded, just like they did not have to be persuaded into their religion, culture etc. They just accepted the idea without much thought, which does not equate persuasion.

That's a bizarre take.

Of course tens fmillions needed to be persuaded, and the Nazis did a lot of propaganda, even calling that openly, false flag operations, and similar things, and of course there was decades of pamphlets, speeches etc before the Nazis even existed. Antisemitism didn't just grow on trees.

You find it bizarre that people are born into a culture and accept much of it without the need to be persuaded. Have you been to a country with a very different culture than yours? That normally opens our eyes to just how much we take for granted is just cultural, and other people would strongly disagree with even your deepest held belief.

SNobody said otherwise, but that's not applicable here.

We're not talking about general customs or viewpoints a people has as part of their culture.

We're talking about a specific political program developed and propagandized in the span of slightly over a decade. Nazism wasn't merely "the general culture" of Germany since always, nor was "kill all Jews".

And yet AI can't write decent landing page copy, or even a convincing cover letter. Seeing the obtuse "reasoning" from Claude daily, I'm very un-worried about AI being a world-ending persuasion engine.

You only spot the ones that aren't convincing.

>persuasion

Is there evidence that such soft skills are improving over time? I take as evidence to the contrary the fact that LLM writing has not substantially improved. I don't see that its powers of persuasion has either, whether that's through making strong arguments for a course of action or writing persuasive essays.

Only the labs knows the truth, but the question as to how many Americans used ChatGPT/Grok/Claude to help them decide how to vote in the most recent election is hopefully going to be something that historians will be able to unearth later.

I doubt the number is low. Knowing everything I know about llms, I still fall into the same groove as I would if I was talking to a buddy online. That part is not far fetched.

I wonder if we’re approaching a cultural moment where people in general are doing exactly what they did when Google first became ubiquitous—search to confirm their biases—but now with LLMs instead of Google. But where Google would happily feed them some link to potential mis/disinformation, the AI is more likely to say “no, that’s not accurate, and here’s why…”

I'm a physicist, and I will do my own calculations before I buy some solar/batteries in the near future, but I've been using ChatGPT and Claude to come up with potential candidates and ideas/scenarios. There's approximately 0% chance that they are not advertising what their owners are being paid for.

[delayed]

Muh shareholder value. Corporations are misaligned superintelligences.

I remember reading about research on using LLM's for political persuasion. It worked well.

There was also research on using LLM's in reddit to persuade, it also worked well.

I went to a talk by an author of one of these papers (David Rand). IIRC the format of the paper's experiment was collecting people with conspiracy theory beliefs and having LLMs with different strategies try to reduce their confidence in the conspiracy theory.

The best strategy, perhaps reassuringly, was tailoring facts to the person's worldview. I don't mean massaging facts, I mean that if the person thinks 9/11 was an inside job because of something about melting steel, then the LLM will present some sourced facts about how that actually works. And then it will do the same thing for all six of the sub-beliefs that support their overall belief. This was key to the strategy: the LLM has enough breadth and recall to exactly and patiently rebut most mistakes a person is making without making them defensive.

This strategy handily beat other things like flattery, selectively lying, etc. So while LLMs can clearly persuade, the evidence suggests it is easiest for them to persuade belief in something that is actually true.

This was in 2024 or 2025. Things might be different now.

Not even that much persuasion is necessary.

We have the medium of persuasion: it's called money, and people do ALL SORTS of horrible things that they wouldn't otherwise do when incentivized by money.

We also already have the mechanism of a machine incentivizing and directing people with money, and that's called a corporation (the original paperclip maximizers, by the way).

So you just need an LLM CEO of a legal corporation with financial resources -> near unlimited possibility of real-world impacts over time.

[delayed]

No bait-and-switch here. The discussion started with the idea that LLMs are helpful at improving LLMs. The goal of improving an LLM is to make the LLM more "intelligent" (whatever humans think that means). If they're helpful at that, they have to keep becoming more "intelligent".

I didn't present any evidence that more-intelligent means more-persuasive. If you don't think it does, then there is nothing to think about in the point I made, as no matter how "smart" the models get they will likely remain subject to human wills.

> Let's also imagine for the purpose of conversation that one of the things that can be gotten at an extremely high skill level through increased intelligence is the power of persuasion.

Definitely not true as you can't use reinforcement learning to increase that capability. Only LLM capabilities that you can improve in an RL loop grow substantially.

Agreed. Like lots of this AI speculation appears to require the belief that AI will figure out how to build supervised training environments without more data.

I think this is very unlikely, but world changing if true.

Additionally, what does super intelligence even mean and why would we expect it to be obtainable from the data that we have?

I'd be delighted if someone can provide some definitions for these terms.

> Definitely not true as you can't use reinforcement learning to increase that capability.

What? Citation needed.

Well don’t you need a bunch of humans to test your persuasion capabilities? How would you run improvement tests in a loop??

You’d launch something like ChatGPT and get millions of people to use it everyday.

What is the measurement? I don't think most people come back to a conversation and say "gee you really convinced me to do X, here's the proof that I did it" and even if they did that data would still be very messy and require verification.

Uh this seems exactly backwards - reinforcement learning with human feedback teaches only the power of persuasion. The only way to get rewarded is to persuade a human to reward you. Why do you think the models are all so damn sycophantic?

You can't put RLHF in an automated recursive improvement loop because you need humans for it.

LLMs aren't magic, there's nothing they can do to persuade people to do weapons testing, it's an absurd scenario.

The human tendency to want to finish things in a fraction of a second will definitely take us there.

If Putin can persuade/bribe/blackmail people to fly drones around European airports, then why wouldn't an LLM be able to do similar things?

Putin's persuasiveness does not derive from his intelligence.

Except that Putin doesn't do what you listed, he is not smart enough to do it. And this is an unnecessary complication. He just pays people and they do the job. The more money, the more willing people are to commit crimes.

It feels like I don’t understand your definition of “persuade”, because the notion that people cannot be persuaded of things via reports and flashy animations is to me, absurd.

For what value of N do you think there exist a string of N characters I could have responded to your comment with that would have infected you with a stably self-replicating and capability-augmenting memetic virus with human extinction as its latent goal?

If not you, in this situation, with N characters delivered via a HackerNews comment, then who, in what situation, with what media and how much of it?

Never heard of blackmail?

And by the way, LLMs do not have to convince 1 specific person, 1 random person is sufficient.

You're not spelling out a scenario we can meaningfully dissect.

Probably you will agree, the leverage of a blackmail attack is not bottlenecked by the persuasiveness of the accompanying threat letter. It is bottlenecked by the existence of damning facts they would rather keep hidden, and the ability to discover those damning facts. I'll grant that automated discoverability of those damning facts is augmented by superintelligence; they can efficiently sweep the open internet in the blink of an eye, and sure, I'll concede, hack into most accounts at will. But that is still a small part of the process by which blackmail material is actually obtained in the real world.

> LLMs do not have to convince 1 specific person, 1 random person is sufficient

No, not one random person either, but one person in a position of great influence. There already exists blackmail and other corrupting influences on such persons in voluminous quantities. If the US targets the entire Russian admin, or vice versa, with AI-enabled blackmail discovery techniques, it will not be an augmentation of their preexisting operations to discover blackmail on each other in excess of the augmentation provided to software engineering generally by AI.

Your argument does not really read like an argument against my argument.

Not sure what to say.

You did not provide an argument. You avoided answering my direct questions, provided a sarcastic 4-word reply, and so I extrapolated a reasonable rebuttal in its place, and replied to that rebuttal.

I should just have said: Yes, I have heard of blackmail. I remain unconvinced AI will cause human extinction.

Ah yes, the "just do what a smarter person than you would do in this situation" argument. If I had that power, would I be wasting it shitposting here? We know humans have infectible little minds. At their worst, humans are just stoichastic parrots. Throw something obviously AI generated at a stoichastic human-parrot and you'll inevitably get the answer "Why should I read something if you didn't bother to write it?" UBI as a memetic virus comes close tho.

Personally, I've been duped by a good ad into buying something I totally didn't want or need so I admit the possibility that the exact right tailored argument to me at the right time and place would get me to do something I'd regret later. Fortunately I'm not in charge of anything serious/am smart enough to not install software for an interview on my work laptop while working at Binance.

I don't know that there exists some N for which that is true, because people are different, so what would convince me to do something isn't the same aa needed to convince someone else to do that something.

I think that is a fascinating question and I would like to advance the provocative notion that N is much smaller than you think, maybe <256. Consider that 128 bits is a cryptographically secure key, and English text has an entropy of around half a bit to a bit per character. 256 characters is sufficient to encode sentences that have never been seen or imagined before, totally unguessable sentences. Who knows what's in that probability space?

Do we have reason to believe they would be successful at other extreme feats of persuasion such as persuading a financial institution to transfer all of its holdings to an account controlled by the AI or persuading the Pope to renounce God?

> Let's also imagine for the purpose of conversation that one of the things that can be gotten at an extremely high skill level through increased intelligence is the power of persuasion. A "superintelligence" persuades humans to perform weapons tests and then to give it the ability to use the weapon.

Is there any evidence, either by way of quantifiable data or by step-by-step thought experiment, that suggests this is possible?

I consider it supremely silly, for example, that there is a graph of destructiveness as a function of persuasiveness where "y = manifest human extinction" is reached at some finite level of persuasiveness, as though persuasion is a single scalar that grows without bound, and the only safeguard between a conversation and WWIII is just a mind waiting for the right argument as expressed in a ChatGPT transcript. (Keep in mind, also, that you have already ceded to the OP that autonomous intellectual and physical self-improvement at singular trajectories is implausible.)

IMHO, one of the top-p vectors is probably something similar to what we've already witnessed with HuggingFace, (e.g. as part of an eval or RL run for cyber; things like Daybreak and Mythos are product lines), just worse in scope and somehow eluding monitoring again.

You don't need a super intelligence for that. You just need a malicious or greedy person with an imperfect AI.

The super intelligence probably already knows that conflict is meaningless and a human flaw.