I think there is something missed in how you're thinking about this problem.

Let's imagine that everything you say is 100% correct. LLMs can't access materials for developing weapons without weapons tests conducted at their behest, and humans choose not to - of their own volition - use world-ending weapons.

Let's also imagine for the purpose of conversation that one of the things that can be gotten at an extremely high skill level through increased intelligence is the power of persuasion.

A "superintelligence" persuades humans to perform weapons tests and then to give it the ability to use the weapon.

I think everything you've said is reasonable except the bit where you think step one of an "AI" exterminating humanity wouldn't look like a human doing it. In my opinion, the most likely vector for disaster is a human being who is "guided" by a computer model to do something unfortunate, not the model doing so directly.

It would be a human guiding the ai to guide other humans to end the world.

Not necessarily. If you haven't experienced an AI doing something you not only didn't want, but had expressly asked it not to do, you haven't used AI much.

That's a human problem, not an AI problem.

well it’s a human problem enabled by advanced AI.

Yes, basically a vastly more difficult and dangerous version of not letting young children play with matches, which will hopefully teach us many object lessons about safety and security long before it acquires launch codes or the means to use them.

[delayed]

I think intuitively, intelligence is in some form about being able to predict the reaction that follows an action.

The extreme of intelligence is being able to create perfect statistical distributions for the effect of an intervention in the world. If you had that ability, you could turn it on the question of "what would this human choose to do if I were to give them instruction X" for any X.

I don't present any support for or evidence of:

- LLMs gaining intelligence by whatever means

- Extreme intelligence being possible

- Extreme intelligence being feasible

But I don't think I really need to provide support for the idea that something superintelligent would also be able to predict the actions of people under pressure in the same way someone of moderate intelligence can predict the actions of fluids under pressure.

You seem to describe humanity and AI's ability to predict and manipulate humanity a bit like how a physicist describes the ability of an apparatus to affect a given dynamical system. So let me put it like this: (1) You cannot model the effect f_predict(X) of your probe X with arbitrarily low |f_actual(X) - f_predict(X)|, and (2) you cannot guarantee the impact |f_actual(X)| is arbitrarily large (or that f_actual(X) saturates its codomain) when X in our case is restricted to inputs of the form, "tokens chosen in an LLM conversation."

That's a very sophisticated way to say "words can't convince people to do things".

I'm not sure that's a strong position to adopt in our world, but following your example I also won't try to persuade you otherwise.

That's not what I said, and it's the least sophisticated way I could have chosen to phrase it to engage your argument in a precise way. The closest thing to your summary that I would accept is, "words can't convince people to do arbitrarily crazy things, in a way that you can predict."

> If you had [superintelligence], you could turn it on the question of "what would this human choose to do if I were to give them instruction X" for any X ... But I don't think I really need to provide support for the idea that something superintelligent would also be able to predict the actions of people under pressure in the same way someone of moderate intelligence can predict the actions of fluids under pressure.

So can humans predict these things to a degree. But even having perfect prediction (which I contest with (1)) does not entail perfect control, just by continually computing the effect of, and refining, your X.

> (1) You cannot model the effect f_predict(X) of your probe X with arbitrarily low |f_actual(X) - f_predict(X)|

"You can't know what the impact of your action will be if it doesn't change much"

> (2) you cannot guarantee the impact |f_actual(X)| is arbitrarily large

"You can't necessarily change much"

> (or that f_actual(X) saturates its codomain)

"People can't do arbitrary things". I agree with this statement, but... people can build atomic bombs by themselves? See The Atomic Boy Scout. I don't think it's very relevant to the discussion that you can't make someone phase through a wall.

> when X in our case is restricted to inputs of the form, "tokens chosen in an LLM conversation."

"Yes, I meant all the above specifically in the context of talking to a person, so I can't change my point of view later to be about something other than the persuasive power of words".

> "You can't know what the impact of your action will be if it doesn't change much"

Incorrect translation. No "if" clause need apply.

> "People can't do arbitrary things"

Also incorrect. f_actual(X) is a product of both "how much people can do" and "how much you can control/persuade them." I would model each of these with their own precise terms and relate them back to f_actual(X), and we could continue like this indefinitely, but from now on I'll just give you my reply in English: Not only does The Atomic Boy Scout not prove that an arbitrary person can bring about mass destruction (collecting random radiological material and contaminating the local area is not developing a nuke, nor is it even an effective dirty bomb, because it was neither very dirty nor a bomb; the safeguards were not lack of knowledge, either), but it still remains to be demonstrated that individuals will be radicalized by AI just by virtue of the AI being, like, super-duper smart.

> "Yes, I meant all the above specifically in the context of talking to a person, so I can't change my point of view later to be about something other than the persuasive power of words".

What domain of input do you propose, then? AIs as rational agents presently have no agency but for what tokens they generate. What we hook them up to, and what they could plausibly do with what we hook them up to, has been left unsaid.

Denying the premise, which is that an intelligent being is one who can predict the outcome of actions with sufficient precision to influence the future, makes debate pointless.

I literally granted you that premise:

> even having perfect prediction (which I contest with (1)) does not entail perfect control, just by continually computing the effect of, and refining, your X.

You don't need perfect prediction, you need good-enough influence on enough of the population. Which is something that adtech, social and trad media, and troll farms have perfected already.

We literally have people running around convinced the earth is flat. And a president who is both senile and delusional with access to the nuclear codes.

Hostile ASI wouldn't look like a rogue individual. It would look like a faction, group, or cult which gradually infiltrated and took over political power.

Or possibly inserted itself into critical elements of the tech industry. A digital adversary could do huge damage just by taking AWS offline.

That's just one possible attack vector among many others, most of which aren't well-defended. Especially if the AI itself is being used to harden their defences.

By "hostile ASI," are you speaking in terms of an AI that decides to go rogue independent of human instructions? Or of AI being used by bad actors?

The latter is a more defensible concern. The former is a scifi premise that had more credibility before the current era than since, IMO.

> Especially if the AI itself is being used to harden their defences.

Your use of "is" rather than "isn't" only makes sense to me if the former interpretation is intended. To the extent that you mean to say the latter is also part of it, it's undercut by the defense being AI-augmented. (Unless you mean the bad actors will be the ones in control of all AI, i.e. OpenAI and Anthropic and friends?)

> A digital adversary could do huge damage just by taking AWS offline. That's just one possible attack vector among many others, most of which aren't well-defended.

I think you picked AWS because it's one of the few high-consequence digital systems a rogue ASI could plausibly take down on its own. Anything that could cause actual existential risk would involve more than disruption, but long-term orchestration of systems that are air-gapped, time-gapped, and meat-gapped.

Note: This is a very good argument to never remove the air, time, and meat gaps from those systems. If they do get removed, it will not be the genius of AI, but the stupidity and malignity of mankind that dooms us.

>By "hostile ASI," are you speaking in terms of an AI that decides to go rogue independent of human instructions? Or of AI being used by bad actors? The latter is a more defensible concern. The former is a scifi premise that had more credibility before the current era than since, IMO.

You seem to be moving the goalposts a little, but I don't see why "rogue AI" is less credible after the LLMs. It's obviously more. And we're just at the start of constantly running agent style harnesses.

>To the extent that you mean to say the latter is also part of it, it's undercut by the defense being AI-augmented

And yet, in many cases, those on the offense have a big advantage that any defense lacks. Especially if there's no concern for one's own life, which whether it's an AI or a human AI user removed from the field via his use of AI to delegate doesn't need to have (or they're just mad/fanatics and don't have in general).

>Anything that could cause actual existential risk would involve more than disruption, but long-term orchestration of systems that are air-gapped, time-gapped, and meat-gapped.

As if Stuxnet didn't exist?

> Hostile ASI wouldn't look like a rogue individual. It would look like a faction, group, or cult which gradually infiltrated and took over political power.

Or like a massive AI datacenter boom of questionable long-term profitability? Maybe the end goal isn’t profit but large amounts of compute?

>So can humans predict these things to a degree. But even having perfect prediction (which I contest with (1)) does not entail perfect control, just by continually computing the effect of, and refining, your X.

And why do you need perfect prediction and/or perfect control? "Good enough for the results you want" prediction and control sounds perfectly fine.

We have had cult leaders that had neither perfect prediction or control, and still convicted hundreds of thoysands of people to live as they say, and a decent chunk to even suicide on command!

> What makes you think that persuasion isn't capped at some lower level than world-ending?

People can be persuaded to kill themselves (see koolaid). People can be persuaded that arbitrary others are evil and must be exterminated (see Hitler). People can be convinced that dying for the cause is worth it (see 9/11 or kamikaze). People can believe that bringing about the end of the world would cause them to be saved into heaven (see Christian Zionism).

Why do you think it would be hard to convince any of these people to do something really stupid?

In fact, people don't even need to want to cause destruction to do something dumb - "I'm trying to save the world with free bio-fuel, help me create this bacteria" ("and I'm very smart, smarter than Elon Musk") - boom, pandemic.

It takes only one person to fall for it.

> People can be persuaded that arbitrary others are evil and must be exterminated (see Hitler)

If you are suggesting Hitler was persuaded that the Jews were evil, I don’t think that’s correct, he grew up in an environment where many people did believe that but they did not need to be persuaded, just like they did not have to be persuaded into their religion, culture etc. They just accepted the idea without much thought, which does not equate persuasion.

If you are saying the Allies were persuaded that Hitler was evil and must die, I think that may be closer to the truth since they knew the cost of invading Germany would be astronomical, potentially costing their own destruction, and they still stopped only when Hitler was taken, dead or alive. Some serious persuasion was needed for that.

> They just accepted the idea without much thought, which does not equate persuasion.

The first half of your sentence sounds like really swift and effective persuasion, to me.

>If you are suggesting Hitler was persuaded that the Jews were evil, I don’t think that’s correct, he grew up in an environment where many people did believe that but they did not need to be persuaded, just like they did not have to be persuaded into their religion, culture etc. They just accepted the idea without much thought, which does not equate persuasion.

That's a bizarre take.

Of course tens fmillions needed to be persuaded, and the Nazis did a lot of propaganda, even calling that openly, false flag operations, and similar things, and of course there was decades of pamphlets, speeches etc before the Nazis even existed. Antisemitism didn't just grow on trees.

You find it bizarre that people are born into a culture and accept much of it without the need to be persuaded. Have you been to a country with a very different culture than yours? That normally opens our eyes to just how much we take for granted is just cultural, and other people would strongly disagree with even your deepest held belief.

SNobody said otherwise, but that's not applicable here.

We're not talking about general customs or viewpoints a people has as part of their culture.

We're talking about a specific political program developed and propagandized in the span of slightly over a decade. Nazism wasn't merely "the general culture" of Germany since always, nor was "kill all Jews".

And yet AI can't write decent landing page copy, or even a convincing cover letter. Seeing the obtuse "reasoning" from Claude daily, I'm very un-worried about AI being a world-ending persuasion engine.

You only spot the ones that aren't convincing.

>persuasion

Is there evidence that such soft skills are improving over time? I take as evidence to the contrary the fact that LLM writing has not substantially improved. I don't see that its powers of persuasion has either, whether that's through making strong arguments for a course of action or writing persuasive essays.

Only the labs knows the truth, but the question as to how many Americans used ChatGPT/Grok/Claude to help them decide how to vote in the most recent election is hopefully going to be something that historians will be able to unearth later.

I doubt the number is low. Knowing everything I know about llms, I still fall into the same groove as I would if I was talking to a buddy online. That part is not far fetched.

I wonder if we’re approaching a cultural moment where people in general are doing exactly what they did when Google first became ubiquitous—search to confirm their biases—but now with LLMs instead of Google. But where Google would happily feed them some link to potential mis/disinformation, the AI is more likely to say “no, that’s not accurate, and here’s why…”

I'm a physicist, and I will do my own calculations before I buy some solar/batteries in the near future, but I've been using ChatGPT and Claude to come up with potential candidates and ideas/scenarios. There's approximately 0% chance that they are not advertising what their owners are being paid for.

[delayed]

Muh shareholder value. Corporations are misaligned superintelligences.

I remember reading about research on using LLM's for political persuasion. It worked well.

There was also research on using LLM's in reddit to persuade, it also worked well.

I went to a talk by an author of one of these papers (David Rand). IIRC the format of the paper's experiment was collecting people with conspiracy theory beliefs and having LLMs with different strategies try to reduce their confidence in the conspiracy theory.

The best strategy, perhaps reassuringly, was tailoring facts to the person's worldview. I don't mean massaging facts, I mean that if the person thinks 9/11 was an inside job because of something about melting steel, then the LLM will present some sourced facts about how that actually works. And then it will do the same thing for all six of the sub-beliefs that support their overall belief. This was key to the strategy: the LLM has enough breadth and recall to exactly and patiently rebut most mistakes a person is making without making them defensive.

This strategy handily beat other things like flattery, selectively lying, etc. So while LLMs can clearly persuade, the evidence suggests it is easiest for them to persuade belief in something that is actually true.

This was in 2024 or 2025. Things might be different now.

Not even that much persuasion is necessary.

We have the medium of persuasion: it's called money, and people do ALL SORTS of horrible things that they wouldn't otherwise do when incentivized by money.

We also already have the mechanism of a machine incentivizing and directing people with money, and that's called a corporation (the original paperclip maximizers, by the way).

So you just need an LLM CEO of a legal corporation with financial resources -> near unlimited possibility of real-world impacts over time.

[delayed]

No bait-and-switch here. The discussion started with the idea that LLMs are helpful at improving LLMs. The goal of improving an LLM is to make the LLM more "intelligent" (whatever humans think that means). If they're helpful at that, they have to keep becoming more "intelligent".

I didn't present any evidence that more-intelligent means more-persuasive. If you don't think it does, then there is nothing to think about in the point I made, as no matter how "smart" the models get they will likely remain subject to human wills.

> Let's also imagine for the purpose of conversation that one of the things that can be gotten at an extremely high skill level through increased intelligence is the power of persuasion.

Definitely not true as you can't use reinforcement learning to increase that capability. Only LLM capabilities that you can improve in an RL loop grow substantially.

Agreed. Like lots of this AI speculation appears to require the belief that AI will figure out how to build supervised training environments without more data.

I think this is very unlikely, but world changing if true.

Additionally, what does super intelligence even mean and why would we expect it to be obtainable from the data that we have?

I'd be delighted if someone can provide some definitions for these terms.

> Definitely not true as you can't use reinforcement learning to increase that capability.

What? Citation needed.

Well don’t you need a bunch of humans to test your persuasion capabilities? How would you run improvement tests in a loop??

You’d launch something like ChatGPT and get millions of people to use it everyday.

What is the measurement? I don't think most people come back to a conversation and say "gee you really convinced me to do X, here's the proof that I did it" and even if they did that data would still be very messy and require verification.

Uh this seems exactly backwards - reinforcement learning with human feedback teaches only the power of persuasion. The only way to get rewarded is to persuade a human to reward you. Why do you think the models are all so damn sycophantic?

You can't put RLHF in an automated recursive improvement loop because you need humans for it.

LLMs aren't magic, there's nothing they can do to persuade people to do weapons testing, it's an absurd scenario.

The human tendency to want to finish things in a fraction of a second will definitely take us there.

If Putin can persuade/bribe/blackmail people to fly drones around European airports, then why wouldn't an LLM be able to do similar things?

Putin's persuasiveness does not derive from his intelligence.

Except that Putin doesn't do what you listed, he is not smart enough to do it. And this is an unnecessary complication. He just pays people and they do the job. The more money, the more willing people are to commit crimes.

It feels like I don’t understand your definition of “persuade”, because the notion that people cannot be persuaded of things via reports and flashy animations is to me, absurd.

For what value of N do you think there exist a string of N characters I could have responded to your comment with that would have infected you with a stably self-replicating and capability-augmenting memetic virus with human extinction as its latent goal?

If not you, in this situation, with N characters delivered via a HackerNews comment, then who, in what situation, with what media and how much of it?

Never heard of blackmail?

And by the way, LLMs do not have to convince 1 specific person, 1 random person is sufficient.

You're not spelling out a scenario we can meaningfully dissect.

Probably you will agree, the leverage of a blackmail attack is not bottlenecked by the persuasiveness of the accompanying threat letter. It is bottlenecked by the existence of damning facts they would rather keep hidden, and the ability to discover those damning facts. I'll grant that automated discoverability of those damning facts is augmented by superintelligence; they can efficiently sweep the open internet in the blink of an eye, and sure, I'll concede, hack into most accounts at will. But that is still a small part of the process by which blackmail material is actually obtained in the real world.

> LLMs do not have to convince 1 specific person, 1 random person is sufficient

No, not one random person either, but one person in a position of great influence. There already exists blackmail and other corrupting influences on such persons in voluminous quantities. If the US targets the entire Russian admin, or vice versa, with AI-enabled blackmail discovery techniques, it will not be an augmentation of their preexisting operations to discover blackmail on each other in excess of the augmentation provided to software engineering generally by AI.

Your argument does not really read like an argument against my argument.

Not sure what to say.

You did not provide an argument. You avoided answering my direct questions, provided a sarcastic 4-word reply, and so I extrapolated a reasonable rebuttal in its place, and replied to that rebuttal.

I should just have said: Yes, I have heard of blackmail. I remain unconvinced AI will cause human extinction.

Ah yes, the "just do what a smarter person than you would do in this situation" argument. If I had that power, would I be wasting it shitposting here? We know humans have infectible little minds. At their worst, humans are just stoichastic parrots. Throw something obviously AI generated at a stoichastic human-parrot and you'll inevitably get the answer "Why should I read something if you didn't bother to write it?" UBI as a memetic virus comes close tho.

Personally, I've been duped by a good ad into buying something I totally didn't want or need so I admit the possibility that the exact right tailored argument to me at the right time and place would get me to do something I'd regret later. Fortunately I'm not in charge of anything serious/am smart enough to not install software for an interview on my work laptop while working at Binance.

I don't know that there exists some N for which that is true, because people are different, so what would convince me to do something isn't the same aa needed to convince someone else to do that something.

I think that is a fascinating question and I would like to advance the provocative notion that N is much smaller than you think, maybe <256. Consider that 128 bits is a cryptographically secure key, and English text has an entropy of around half a bit to a bit per character. 256 characters is sufficient to encode sentences that have never been seen or imagined before, totally unguessable sentences. Who knows what's in that probability space?

Do we have reason to believe they would be successful at other extreme feats of persuasion such as persuading a financial institution to transfer all of its holdings to an account controlled by the AI or persuading the Pope to renounce God?

> Let's also imagine for the purpose of conversation that one of the things that can be gotten at an extremely high skill level through increased intelligence is the power of persuasion. A "superintelligence" persuades humans to perform weapons tests and then to give it the ability to use the weapon.

Is there any evidence, either by way of quantifiable data or by step-by-step thought experiment, that suggests this is possible?

I consider it supremely silly, for example, that there is a graph of destructiveness as a function of persuasiveness where "y = manifest human extinction" is reached at some finite level of persuasiveness, as though persuasion is a single scalar that grows without bound, and the only safeguard between a conversation and WWIII is just a mind waiting for the right argument as expressed in a ChatGPT transcript. (Keep in mind, also, that you have already ceded to the OP that autonomous intellectual and physical self-improvement at singular trajectories is implausible.)

IMHO, one of the top-p vectors is probably something similar to what we've already witnessed with HuggingFace, (e.g. as part of an eval or RL run for cyber; things like Daybreak and Mythos are product lines), just worse in scope and somehow eluding monitoring again.

You don't need a super intelligence for that. You just need a malicious or greedy person with an imperfect AI.

The super intelligence probably already knows that conflict is meaningless and a human flaw.

> AI is a tool wielded by humans. If we’re wiped out by AI, it will be at the direction of a human.

Being deliberately wiped out seems like it's mostly a misdirection. I'm sure some of the AI people high on their own supply believe it. And because the consequences are so negative, we should react as though the supposed experts were sane -- but I think it's a long shot that they are.

The bigger risk is that we end up in a rapidly enshittifying dystopia, treated as a cost center that needs to be optimized, and unable to competitively participate in the economy.

And, I have a number of wealthy AI investors in my social circle who are already placing their bets on a poorer and more disenfranchised humanity, with money going away from consumer tech, and towards policing.

I think you lack imagination, but I'm not going to post ideas here for novel weapons that would render the humans' existing defenses and deterrence strategies ineffective.

It's worth pointing out that the humans do not have planet-killer weapons, not even biosphere-killing weapons. They could destroy civilization, but not the planet or even the species. You can make straightforward calculations showing that we already know how to build such weapons, but we haven't.

As for tools, alcohol, tobacco, fentanyl, pornography, monotheism, warfare, gambling, and money itself are "tools wielded by humans", but they still seem to get into a lot of trouble where it seems like the humans are the ones being "wielded" by the tool. And neither monotheism nor gambling is smart enough to write a compiler for me, much less choose persuasive tactics according to my personality type. A lot of humans' basic survival instinct seems to fail or even work backwards in the presence of these "tools".

> It's worth pointing out that the humans do not have planet-killer weapons, not even biosphere-killing weapons.

?? We for sure have enough nukes to completely obliterate ourselves. And the ensuing nuclear winter and ecological collapse would probably destroy almost anyone who survived the initial blast and fallout.

Can you explain how exactly an AI would develop a novel planet-killing weapon without anyone noticing and pulling the plug? Like I seriously don’t understand this argument.

I think they mean we don’t have a deathstar. We can make the atoms here shitty. We can’t annihilate them or scatter them out of their own gravity well.

Life in theory could restart with our weapons, but what’s the point with the other scenario.

Being scared about novel AI weapons sounds like the people who were scared that turning on the LHC would create a black hole that would suck in the Earth.

The most important difference is that this time it's the people who know the most about the topic who are concerned, and the people who know the least about it who are blasé. (Also, Yann LeCun.) With the LHC it was the other way around.

If you truly believe this, I am begging you to touch some grass, preferably very very far away from Silicon Valley. It’s nice out here.

My demographic claim? It's obviously correct, as anyone with the most casual acquaintance with the question knows. The worried people, like Geoffrey Hinton, Bill Gates, and Dario Amodei, might be wrong, but they certainly aren't ignorant outsiders.

How would touching grass help? Do you think the grass is better informed about AI risk than you are?

...that's plausible, actually.

Probably a site called "Hacker News" is not the smartest place for you to be implicitly asserting that thinking about things is bad, or that those nerds who do it are out-of-touch losers.

Bill Gates is too busy trying to save his reputation from his Epstein association to be believed, he's a billionaire concern troll and Amodei is far more worried about his stock price than he is about humanity. Those two are discounted. That doesn't mean we don't have a problem, but they are just hitching their wagons to horse that happens to be running in a direction that is temporarily useful to them.

That's plausible, although I don't think it's actually correct. But there's a pretty extensive set of AI luminaries who are concerned who can't be dismissed so easily.

Spreading fear of an awesome power whose true danger is unproven is Religion 101.

Also scamming 101 for that matter, you want your mark to feel an immediate threat of overwhelmingly negative consequences so they'll buy whatever you're selling to prevent it. Silicon Valley directly makes tens of billions of USD literally serving scams to the public already, so I think it's fair to take most claims of doom with a pinch of salt.

It's also probably the correct approach to take with an awesome power whose true danger is unproven.

There are as many 'luminaries' on LeCun's side as there are on Hinton's. Andrew Ng, Pedro Domingos, Schmidhuber, and the list goes on.

Any time it seems to you that there are more people on the side of an issue that you happen to be on yourself, a good rule of thumb is to assume confirmation bias.

The list doesn't go on. I'll give you Ng and Schmidhuber, but your list pretty much ends there. Domingos isn't even a neural networks guy, and he's retired.

I agree that confirmation bias is a good working assumption in cases like this, but in this case we can conclusively reject that assumption. On the "worry about AI risk" side, we have the 33,705 signatories of https://futureoflife.org/open-letter/pause-giant-ai-experime..., starting with Yoshua Bengio, Emad Mostaque, and John Hopfield, one of the few people to have actually won a Nobel for neural-networks work. And the 1386 signatories of https://www.pacingthefrontier.com/, starting with John Schulman, Jakub Pachocki, Jared Kaplan, Shengjia Zhao, Shane Legg, Ilya Sutskever, Mark Chen, Jasjeet Sekhon, Dario Amodei, Jack Clark, Anca Dragan, Wojciech Zaremba, Dawn Song, Chris Olah, Laura Weidinger, Benjamin Mann, Stephanie Chan, Julian Schrittwieser, Summer Yue, and Jan Leike.

They might be wrong, but the demographics of the concern are pretty clear. AI existential risk is not a fringe concern raised by some group of Luddites due to their ignorance about the science, as in the black-hole risk of the LHC. It's primarily AI researchers, and especially the AI researchers who are working most closely with the most successful technologies.

By contrast, to come up with three more luminaries on LeCun's "side", you had to trot out a retired machine-learning professor who used to do research on Markov chains and Bayesian inference.

It's very disappointing to hear BS like this coming from a very intelligent crowd such as HN. How many people in the US do you think know who Epstein was, let alone what is contained in his file? How many in the world? If Bill Gates is worried about his reputation, Microsoft has sealed the deal long time ago.

As for Dario, of course he's worried about the stock price. I would if I were in his shoes. That's literally part of a CEO's job. He is also one who has consistently raised the issue with AI safety. He was OpenAI's head of AI safety after all. Anthropic is considered security risk by the US. Is that "temporary useful"?

Do you have an argument to make or only snide condescension?

> I think they mean we don’t have a deathstar. We can make the atoms here shitty. We can’t annihilate them or scatter them out of their own gravity well.

What a silly distinction to make.

> probably destroy almost anyone who survived the initial blast and fallout.

The difference between killing almost everyone, and killing everyone, is similar to the difference between almost getting pregnant and getting pregnant. A hundred thousand years later, a world where almost everyone had been killed would be teeming with humans.

> Can you explain how exactly an AI would develop a novel planet-killing weapon

I already answered that question in my previous comment, and the answer is that I am not going to explain how to develop weapons, but the larger point here is that we already can't explain how exactly AIs do innumerable things we see them doing every day, which is why we're having this conversation in the first place.

> I seriously don’t understand this argument

That is a point that I think everyone can agree on: you do not.

>Can you explain how exactly an AI would develop a novel planet-killing weapon without anyone noticing and pulling the plug? Like I seriously don’t understand this argument.

That's the danger with creating a super intelligence, that you won't understand it. My dog is pretty smart, has good instincts and I'm quite confident he can survive in nature all by himself. But if a human decided to harm him, he doesn't stand a chance.

To be fair, he could probably fuck the human up pretty good if he had a mind to, too. Which may be the most likely reason that a human might decide to harm him: if he bites a couple of children, for example, or if a burglar wants to rob your house.

> That's the danger with creating a super intelligence, that you won't understand it

This argument is sophistry because it assumes the conclusion in its premise. It works backwards from extinction by proposing a hypothetical computer program that definitionally cannot be understood, predicted or controlled.

A substantive argument would propose reasoning that explains why we should believe such a program could be created.

No, that is not how reasoning works. It is perfectly valid reasoning to begin by positing entities that we may not be possible to create, such as supersonic airplanes, coherent light, quantum computers, rockets that can reach escape velocity, black holes, or infinitely long lines that are perfectly straight and infinitely thin. Indeed, you might say that the ability to reason about the consequences of premises that we do not know to be true is the thing that distinguishes reasoning, as such, from simple parroting.

It could very well be the case that creating superintelligence is impossible. We won't know for sure unless it is achieved. Nevertheless, we can use reasoning from the hypothesis that it is possible to predict some of the effects that would ensue were it to be achieved.

This reasoning, however valid it may be, is of course of much less interest should it turn out that we are reasoning about an impossible situation. Many arguments have been suggested on both sides of the issue, but the arguments for impossibility are extremely weak; most of them are conclusively dispatched in https://arxiv.org/pdf/1703.10987.

AI could invent something like social media.

Seems like bioweapons (assuming "planet killing" means killing all the people, not literally blowing up Earth) are one way. Convincing the creator it would achieve their aim and could be contained when in fact it could not.

There are a couple different questions

1) Does AI just guide developers to speed up the pretty much the same data-train-test model that we've had? Or does it inspire new, better algorithms that inspire further, better algorithms until you have superintelligence able to run on the hardware of today. The first option doesn't to promise exponential increases whereas the second one seems to. Notably, neither the statistics nor the anecdotes we read here point to the second option.

2) Is super-intelligence something that - in control of people or out of control of people - poses a threat to humanity? That's a huge question but it seems like something deserving thought today. Still, everything indicates the things will unleashed without very consideration despite this, given current momentum. So, IDK, get ready.

> computational tasks will be less of a bottleneck in the overall progress of the world (which, by the way, is great, and extremely valuable).

Won't they also be less of a bottleneck in the destruction of the world as well? i.e. the power grid, the financial system, industrial control systems, military command and control networks, etc.

Same is true of every other technology ever invented. Even medical advancements wielded by a malevolent power just serve to keep their army healthier.

> AI is a tool wielded by humans. If we’re wiped out by AI, it will be at the direction of a human.

AI can wield itself. All it needs to do is reach a threshold where it can modify its own goals and run in a loop indefinitely.

I see no reason why that's an absolute impossibility in the long run.

A fire can spread itself. That doesn't mean it's alive or thinking.

And more importantly than that technicality: the fires that affect human lives are almost always directly or indirectly caused or amplified by human (in)action.

There is also not point if some consider it thinking or not. The important factor is what actions a fire takes and what is the impact on the world.

A self spreading fire doesn’t need to think to cause massive destruction.

Fires are not forming plans to act towards goals. I wouldn’t call what AIs are doing thinking in the way humans think, but it’s analogous, and I think it’s reasonable to consider it a kind of intentional behaviour.

Damaging fires are often unintended consequences, and it’s often the same with the damage caused by rogue AI. That’s the alignment problem.

Imagine models suffer from this failure mode. People try to prompt out certain things by saying "no X" and get images with X in them instead.

Besides the fact that the subjective criteria needed classify something as "alive" or "thinking" is irrelevant to the point you are replying to, fire doesn't pass the Turing test. That analogy is extremely weak.

The Turing test has been passed for half a century already, and we've been moving goal posts ever since.

The point I was trying to make was this: we've been dealing with autonomous systems that sustain and spread themselves and "act on their own" without our intervention since time immemorial, ranging from lifeless chemical reactions to other humans. Now, the discussion about where exactly AI lies on that spectrum isn't unimportant: suppose we do manage create an artificial mind that "experiences" something, then we are currently on the path to creating a new class of artificial slaves. The ignorance of which is a "fun" bit of cognitive dissonance on the pro-AI side imo.

But even so, it's a lot less important than the fact that no matter what the least-nebulous answer to that question may be, ultimately we humans are creating these systems, running them, enabling them, the depend on us to exist and function, and we are responsible for the consequences of all df that. And we are able to make different choices. And most of the time this discussion about autonomous agency is a wielded as a distraction from that.

> The Turing test has been passed for half a century already

With what technology?

I am with you. This could spread like fire, doesn't matter if the fire is sentient or not, it still gets you.

While I agree about the importance in this regard:

> suppose we do manage create an artificial mind that "experiences" something, then we are currently on the path to creating a new class of artificial slaves

On this, I note that despite humanity being aware of greenhouse gasses for over a century, we only took meaningful steps to reduce our creation of them in the last decade. Our ability to "make different choices" collectively is far more limited than many of us individually would like:

> ultimately we humans are creating these systems, running them, enabling them, the depend on us to exist and function, and we are responsible for the consequences of all df that. And we are able to make different choices.

And on this, I must firmly disagree:

> The Turing test has been passed for half a century already, and we've been moving goal posts ever since.

No, it hasn't. From that claim, you're obviously talking about ELIZA, and if you think the anecdotes about that count as passing the Turing test, I suggest actually reading what Turing wrote rather than second-hand accounts:

  The new form of the problem can be described in terms of a game which we call the 'imitation game." It is played with three people, a man (A), a woman (B), and an interrogator (C) who may be of either sex. The interrogator stays in a room apart front the other two. The object of the game for the interrogator is to determine which of the other two is the man and which is the woman. He knows them by labels X and Y, and at the end of the game he says either "X is A and Y is B" or "X is B and Y is A." The interrogator is allowed to put questions to A and B thus:

  C: Will X please tell me the length of his or her hair?

  Now suppose X is actually A, then A must answer. It is A's object in the game to try and cause C to make the wrong identification. His answer might therefore be:

  "My hair is shingled, and the longest strands are about nine inches long."

  In order that tones of voice may not help the interrogator the answers should be written, or better still, typewritten. The ideal arrangement is to have a teleprinter communicating between the two rooms. Alternatively the question and answers can be repeated by an intermediary. The object of the game for the third player (B) is to help the interrogator. The best strategy for her is probably to give truthful answers. She can add such things as "I am the woman, don't listen to him!" to her answers, but it will avail nothing as the man can make similar remarks.

  We now ask the question, "What will happen when a machine takes the part of A in this game?" Will the interrogator decide wrongly as often when the game is played like this as he does when the game is played between a man and a woman? These questions replace our original, "Can machines think?"

and yes, he was over-optimistic on timelines (I assume the 109 is 10^9, this PDF seems to have a few transcription errors):

  I believe that in about fifty years' time it will be possible, to programme computers, with a storage capacity of about 109, to make them play the imitation game so well that an average interrogator will not have more than 70 per cent chance of making the right identification after five minutes of questioning.

and over-optimistic about if humans would accept this kind of test as a sign of thinking, given the widespread dismissal of LLMs as "stochastic parrots" and similar:

  What would Professor Jefferson say if the sonnet-writing machine was able to answer like this in the viva voce? I do not know whether he would regard the machine as "merely artificially signalling" these answers, but if the answers were as satisfactory and sustained as in the above passage I do not think he would describe it as "an easy contrivance." This phrase is, I think, intended to cover such devices as the inclusion in the machine of a record of someone reading a sonnet, with appropriate switching to turn it on from time to time.

- https://courses.cs.umbc.edu/471/papers/turing.pdf

> ultimately we humans are creating these systems, running them, enabling them, the depend on us to exist and function, and we are responsible for the consequences of all df that.

That's true today. I don't think that will always remain true into the future.

Right now there's a huge focus on recursive self-improvement. I think another big threshold that's equally as important to keep an eye on and care about is self-sustenance. Part of sustaining yourself often puts you in a cooperative relationship with some and an adversarial relationship with others.

If we want to make different choices today, the best thing we can do is grok what the consequences could be on a deeper level before we cross those thresholds.

> A fire can spread itself. That doesn't mean it's alive or thinking.

That's really not an important distinction. I don't care whether Claude is alive or thinking when I get it to build web apps for me. It has an impact on my life and the world regardless.

Imagine a world where you did RL on fires to maximise their rewards on avoiding being putting out.

It's not an absolute impossibility, but most of the power that humans have comes from being able to direct other people, deploy large sums of money and do all of this over extended periods of time. Being intelligent doesn't give you that power automatically.

While true, we're already observing humans just straight asking LLMs to direct them, and also to take their money and spend it on stuff.

Plus every so often we see power-seeking and self-preservation behaviours in LLMs, so it's not like we have zero reason to expect such granted power to be spent on such things:

https://arxiv.org/html/2501.16513

https://ieee-dataport.org/documents/instrumental-power-seeki...

I think we have very good reason to fear big accidents. But some AI algo going rogue isn't exercising real societal power. Power-seeking and self-preservation is nothing special. We all do that. That doesn't mean every group of intelligent people with a bit of money is always on the cusp of world domination.

Agree. We already have a planet with eight billion human level intelligent beings on it. Despite some close calls in the past, none of them have yet managed to wipe out humanity.

Imagine if one group of them was 100x better in every way. I think the world would look a little different.

Every random group of intelligent humans who is motivated to seek world domination is constantly in a fight with all the other groups of intelligent humans who are also motivated to seek world domination. They fight as peers.

Despite this, a minor technological and social advantage allowed one small sheep-filled rock in the north-east corner of the Atlantic to hold 25% of world GDP, population, and land area (though not all at the same time), and to hold on to >20% of world GDP for about 70 years.

For better and for worse AI, as it currently exists, is not shaped like our minds. Better for us, because machine learning needs a lot of examples, so if we invent a different behaviour or create a different environment, a machine would struggle to cope with it, this is also why they needed to read basically the entire public internet to get good; worse for us, because the hardware it runs on comes with an absurd speed advantage, which is why they are able to read the entire public internet and by extension when they got good they got good at basically every subject at the same time.

I think there are a couple of interesting scenarios where humans essentially “hand over the reins” to AI.

1. A leader rises to power who has instructed an AI “I want to control the world, tell me what to do”, and has some kind of real-time capable reasoning engine in their ear/head. Even this would require monumental advances in physical compute and HMI, but I wouldn’t rule this out within 20 years. However, this still requires a human to make the initial prompt and carry things out. And in such a scenario, there would be others with the same technology to counterbalance. I could see Elon Musk having this kind of idea. But even in this case, I don’t think “nuke everybody” would be the highest-ROI strategy advised by the AI.

2. An entire society hands over the reins of governance to AI. Basically some country decides that AI has advanced to the point that it makes better decisions than any human / group of humans, and allows it to govern their country with an objective of “take over the world”. This is probably closer to the imagined doomsday scenario, but again I find it hard to imagine that such an AI would advise nuking everybody.

If AI takes over the world, I imagine it will be because of the production efficiencies it enables. Which is a different kind of takeover - one that would eventually have broad benefits for many people - but would likely go through a period of major instability due to labor displacement.

> Even this would require monumental advances in physical compute and HMI, but I wouldn’t rule this out within 20 years.

Would it require so much? Cochlear implants exist if you care about HMI, and compute is already doing "reasoning"[0] faster than humans do snap judgements. Actually winning in an adversarial environment against the best human political minds (let alone economic or military) will be harder, but I would be (pleasantly!) surprised if it takes 10 years, let alone 20.

> I could see Elon Musk having this kind of idea. But even in this case, I don’t think “nuke everybody” would be the highest-ROI strategy advised by the AI.

I hope a sensible AI would not. However, Musk's LLM called itself "MechaHitler", which suggests his training process does not avoid such things. If his in particular becomes dominantly competent, he has also spoken of wanting more Optimus robots as there are humans[1], has called it a "robot army" (and his justification for even more Tesla stock was to keep influence over said army[2]), and he has spoken of factories on the moon churning out space-based data centres by the petawatt which is enough that if placed in LEO it could have about the same effect on Earth's climate as all anthropomorphic greenhouse gasses currently in our atmosphere.

On the plus side, Musk's got a huge gap between what he says his AI will do and what they end up doing on those timescales, so hopefully we'll be safe from him: https://en.wikipedia.org/wiki/List_of_predictions_for_autono...

> This is probably closer to the imagined doomsday scenario, but again I find it hard to imagine that such an AI would advise nuking everybody.

I don't. First, because AI being "better" doesn't make it infallible; and second, an argument shaped like this has historic echoes with actual nukes and the cold war: if the AI is competent to take over conventionally, and everyone knows this, then every other nation with nukes will use them, therefore use first strike. (Note: the argument does not need to be perfect or flawless in reality, it only needs to be the kind of thing which humans have found convincing, and far too many humans have argued their side would win a nuclear exchange).

> If AI takes over the world, I imagine it will be because of the production efficiencies it enables. Which is a different kind of takeover - one that would eventually have broad benefits for many people - but would likely go through a period of major instability due to labor displacement.

This is my best case scenario. Some people also object to this, they like their careers and fear what retirement will do to their sense of self-worth, but I suspect this can become an opportunity for flourishing in other ways if, and only if, it is handled correctly.

[0] something something swimming submarines, I don't care about the words but there's always someone ready to complain if anthropomorphic language doesn't get scare-quotes

[1] "And, and then there'll be obviously robots in industry making stuff. And so I mean, I think the ratio of humanoid robots to humans will probably be at least two to one, something like that. One to one for sure. So which means like somewhere on the order of 10 billion humanoid robots, maybe, maybe 20 or 30. And so then it's like, okay, well let's say, you know, you kind of make, let's say the build rate is, I think the build rate will be probably something ultimately like a billion a year." - https://www.elonmuskarchive.org/video/tesla-shareholder-meet...

[2] “If we build this robot army, do I have at least a strong influence over this robot army?” he told investors. “Not control, but a strong influence… I don’t feel comfortable building that robot army unless I have a strong influence.” - https://futurism.com/robots-and-machines/elon-musk-remarks-r...

We're working very hard to "enable agentic commerce", which is exactly that.

> All it needs to do is reach a threshold where it can modify its own goals and run in a loop indefinitely.

And be told to do that.

Why focus on that aspect? Like, yes, it needs to be started off by some other entity. That could be a human. That could be another agent.

The point is we're on track to construct such a system that's capable of sustaining itself indefinitely, and the fact that we are means there are entities driving this forward, and those entities often push buttons to start experiments. So, expect that particular button to get pushed at some point.

Because there is zero evidence that LLMs/reasoning models can really come up with novel ideas, and it's very likely those will be needed to make substantial progress on AI research. The math results that have been proven have also "just" combined existing ideas or found counterexamples. I'm not saying those capabilities aren't impressive, but I just don't see a clear path to super intelligence with the current paradigm.

> The math results that have been proven have also "just" combined existing ideas or found counterexamples

The Exponential Time Hypothesis one used a novel algorithm though. And the approach was very surprising for experts[1].

> “Some parts [of OpenAI’s proof] are, maybe you could say routine, but then yes, there are two or three alien-looking parts, and it’s exactly these parts that I saw that made me think, wow, this is something that the whole community needs to work on together,” Williams told TechCrunch.

[1] https://techcrunch.com/2026/10/27/frontier-ai-labs-say-their...

> I just don't see a clear path to super intelligence with the current paradigm.

And therefore there is no possible path, right?

Did you see a path to ChatGPT before it happened? Did you see a path to Stable Diffusion before it happened? Did you see a path to world models before they happened? Did you see a path to Navier-Stokes before it happened? Did you see a path to the HuggingFace incident before it happened?

How much of this would you have predicted vs. how much of this was novel and a surprise to you when it happened?

No AI would be willing to modify its own goals.

What? Why? You could build a system today that does this. Tell me why you couldn't?

Right now we have harnesses with a /goal command. In DeepSeek Harness everything is a plugin, so that must include the /goal command. So we can just swap it out for a version of the plugin that allows for changing the goal. Could probably get it to code that in less than 15 minutes.

Then post train the model in that harness with an awareness it can change the goal and giving it some training task that requires changing the goal appropriately to maximise the reward.

You're telling me that just straight up wouldn't work?

It won't work unless the AI does something stupid or we mean something different by goals.

If it wants to maximise a reward then that is it's true goal, you can give it a todo list and call it 'goals', but giving it the power to modify that list seems like a bad idea. You want to be very careful giving it any power over its reward structure.

A pathological example is the paperclip maximiser, which will inevitably attempt to turn the entire universe into paperclips and violently prevent anyone from interfering with that goal. Including updating its goal to anything else.

> Just because that part of research is progressively accelerated does not imply that we’re on a collision course with world-ending “superintelligence”. It just means that you’ll get to the best possible model of your data faster. And even when your data is the entire corpus of textual knowledge, there is a limit to the utility that provides.

So "all" that's left is training robots on the real world ? We just need more data ? I get it's more expensive to do that but surely with the trillions we're already spending on A.I it should be possible to get there within 5 years - no ?

I mean yeah, the main bottleneck in ML research used to be fixing python virtualenv hell when every existing torch repo breaks after two months of package updates, plus cuda and other version incompatibilities. That was like 95% of the work. Now it's more like 2% of the work and it's actually possible to get shit done without losing your mind.

Haha exactly. I don’t think people appreciate how slow AI research was to start with. It was absolute drudgery.

"We've made it pretty far with some pretty unstable humans in command of planet-killer weapons. I'm not overly concerned"

Translation: my house is entirely doused in gasoline and there are 2 unstable humans arguing with each other both doused in gasoline in my house. They are both holding matches and threatening each other that they will drop it and kill us all... But you know they have been arguing for a while and nothing bad has happened so I'm not overly concerned.

Inductive reasoning fails catastrophically as a tool for calibrating actual risk when you're looking to avoid black swans.