This is the type of weak dismissal I’m talking about.

A weak dismissal for a weak argument. The strong dismissal is that many processes, even relatively simple ones, are formally chaotic in the https://en.wikipedia.org/wiki/Chaos_theory sense, meaning that predicting how they evolve over time becomes exponentially more difficult the further ahead in time we try to predict. Meaning we quickly get to the point where all the energy in the known universe wouldn't be sufficient to predict the future. This means AI could never be omnipotent to the degree that LW types seem to believe; no amount of "intelligence" can solve in polynomial time problems that have formally been proven to have exponential complexity, which would severely limit the power of any "superintelligence".

Intelligence needed to be existentially threatening to all human life <<<<<<< intelligence necessary to survey the totality of atoms and particles in the universe. The former is still very possible.

LessWrongers seem to fear a superintelligence that was 95%+ effective at persuading/manipulating humans to achieve whatever its ends were. But this would require the AI being able to predict the future to a degree that wouldn't be remotely possible given the computational power available to it, due to exponentially increasing difficulty.

This is true if and only if human intelligence is anywhere close to any theoretical maximums. I propose an alternate hypothesis: human intelligence is weak and easy to exploit. The only reason it doesn't happen more (and it already happens a lot!) is that we're too stupid to do it reliably.

Consider the amount of compute needed to beat the strongest chess grandmaster that humanity has ever produced, pretty much 100% of the time: a tiny speck of silicon powered by a small battery. That is not what a species limited by cognitive scaling laws looks like.

>This is true if and only if human intelligence is anywhere close to any theoretical maximums

Humans are capable of logical reasoning from first principles. You can fool some of the people all of the time, but no words are sufficient to convince people capable of reasoning to do things that are clearly not in their own self interest.

I'm confused. We know persuasive humans exist, we employ them as politicians or sales people. There are also many unpersuasive humans.

Given we seem to have a decent range of persuasiveness even amongst these very very similar minds, why do you think the upper limit for persuasiveness is a charismatic human?

Though even if that WAS the limit I'd still be somewhat worried due to the possibility of that persuasiveness being used at far greater scale than a human could do...

>Given we seem to have a decent range of persuasiveness even amongst these very very similar minds, why do you think the upper limit for persuasiveness is a charismatic human

Because there's a hard limit on how much people can be made to act against their own self interest.

Is there really? Some religious orgs have gotten quite good at getting people to blow themselves up for the promise of 77 virgins after death.

Ever heard of the Jonestown Massacre? Suicide seems to be a fairly consequential upper limit.

That's like implying it's impossible for computers to be superhuman at chess given the enormous exponential scales of the number of possible positions. Existing highly persuasive humans can create cults of thousands of people willing to follow their every whim. Why is it so hard to perceive of an agent that tirelessly searches over every human bias and tactic to hone a superhuman level of manipulative skill? You don't need to perfectly predict the future to persuade people of things.

The AI can run an A/B test, no? It doesn't need to predict perfectly, it needs a goal and the ability to test and measure, and adapt as things change.

Generative AI creating text/audio/video with a goal and a testing/feedback loop. I'm not an AI and I think I could make it work given a small amount of resources.

Why would you need to predict the future to be convincing? Charismatic people aren't acting like oracles, they're just...being charming. They frame actions in ways that make them look good, do little favours for you, stroke your ego, and otherwise have a good grasp of human social dynamics.

Sure, there's an aspect of understanding the other person, but chaos theory doesn't stop politicians from obtaining and keeping power.

>Why would you need to predict the future to be convincing?

You need to be able to predict how people will react to your words.

Sure, but that's not predicting the future to a high degree of certainty. People predict how others will react all the time. The most convincing humans are already very good at it, so we know it's possible to some degree. While I agree chaos theory puts an upper limit on how well someone can predict the future, there's no particular evidence that human ability is close to the limit.

I think you're imagining some mastermind that has to get everything right or it dies, instead of a master politician or businessperson who's a few percentage points better than the existing competition and for whom the advantage compounds. Yes, Less Wrong is way too hysterical about everything, but charisma that's better than human is a problem.

We genuinely don't know what degree of predictive power would be required to wipe out humans so I'm surprised you're willing to make the bold claim that thisnis impossible.

They don't have to perfectly predict the future, they just have to run faster than we do, unconstrained by a brain that needs to fit in a birth canal or neurons that need chemicals. We had a natural limit to intelligence, for anything more we need to collaborate, often across time and space and while dealing with people who can't stop warring with each other for any number of reasons. It doesn't take much imagination to see how that is beaten. I'd guess a good percentage would be worshiping at the foot af the AI within a few months, so we'd be competing with ourselves in no time.