One of these things is real and happening now. It isn't potential.

The other is a hypothetical future scenario that has literally no A->B->C line drawn to it, pushed by people whose ability to feed themselves depends on attention farming.

So no. Let's not focus on the bullshit one, and let's focus on the one that is hurting people in the now.

So your suggestion is that we ignore the existential risk until after it occurs?

> literally no A->B->C line drawn to it

Spend a couple of hours to educate yourself about the issue, and make a convincing counterargument please:

https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-no...

https://astralcodexten.substack.com/p/why-i-am-not-as-much-o...

https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a...

https://www.youtube.com/@RobertMilesAI/videos

posting links to yudkowsky is just not going to convince people who don't already see it his way. he doesn't even try to be persuasive.

Considering how much progress we've been making with so little understanding of what's happening internally in these AI systems, the concern is that we might get to C rather quickly without anyone ever having drawn out a clear line from A->B->C. Here's an FAQ describing the basic concern:

https://www.lesswrong.com/posts/LTtNXM9shNM9AC2mp/superintel...

This is the most specific worst case scenario I've come across so I'll respond to it:

> For example, it might program a virus that will infect every computer in the world, causing them to fill their empty memory with partial copies of the superintelligence, which when networked together become full copies of the superintelligence. Now the superintelligence controls every computer in the world, including the ones that target nuclear weapons. At this point it can force humans to bargain with it, and part of that bargain might be enough resources to establish its own industrial base, and then we’re in humans vs. lions territory again.

I believe there's a rift between doomers and eye-rollers because this kind of leap sounds either hollywood-hacker-sci-fi or plausible-and-obvious. The notion that software can re-deploy an improved version of itself without human intervention is just outside the realm of possibility to me (or somehow blackmailing or persuading a human to act alongside it?? Is that AI anymore or is that just a schizophrenic who thinks the computer is talking to him?)

I don't think it can be reasonably disputed that software re-deploying an improved version of itself is plausible and obvious. Automated commits and automated deployments are both fairly common in modern build systems.

That software exists to automatically apply changes to code doesn't have much bearing on software being able to introspect and decide what changes to apply, but maybe I'm moving the goalposts. The infrastructure exists for programs to edit their own deployment, what I'm doubting is that the software will have any idea of what can be improved without a human-in-the-loop.

Compiler optimizations are a counterexample, could be called automated self-improvement -- but it's to do the same thing with fewer resources, what I'm looking for is a reason to believe an optimization could go the other way, have more capabilities with the same resources.

It's important to note that the lesswrong's view of AGI is that it sees everything, knows everything, can do everything, can make humans do anything, and has motives directly harmful to us.

These are all taken as a given because the entire concept is just old testament god but with glowy parts. This is an essential part of the dogma, which is why there's never any sort of justification for it. Super smart computer is just assumed to be magic.

It's plausible and obvious to them because "a super-intelligence can make anyone do anything", can reprogram any computer to it's will, and can handwave away literally technical deficiencies.

Replace the term "superintelligence" with "God" in that FAQ. Do you still buy the conclusions?

"A God is a mind that is much more intelligent than any human."

"Go programs had gone from “dumber than children” to “God” in eighteen years, and “from never won a professional game” to “Godlike play” in six months."

"God has an advantage that a human fighting a pack of lions doesn’t – the entire context of human civilization and technology, there for it to manipulate socially or technologically."

"A God might be able to analyze human psychology deeply enough to understand the hopes and fears of everyone it negotiates with."

"Humans instinctively balance thousands of different considerations in everything they do; so far this hypothetical God is only balancing three (least cancer, quickest results, highest probability). To a human, it would seem maniacally, even psychopathically, obsessed with cancer curing. If this were truly its goal structure, it would go wrong in almost comical ways."

(Oh, god, there's my personal bugbear: "bigger brain size correlates with greater intelligence". You know what else correlates with bigger brain size? Hint: I'm 6'5"; how much smarter am I than someone 5'1"?)

Why would replacing 'superintelligence' with 'God' be insightful?

Are you suggesting human intelligence is unique and the upper limit for intelligence? I mean we can argue about the difficulty and timeline for achieving human level artificial intelligence, but it seems unlikely that it's impossible.

No, I'm suggesting the entire hype over "superintelligence" is religious in nature. There is little evidence that the current AI systems are anywhere near human intelligence and none that "more than human intelligence" is possible. Further, the abilities attributed to the "superintelligence" are completely godlike---it's omniscient and omnipotent, right? Nothing it can't do, potentially?

So, if you're not worried about the extinction risk of God, why should you be concerned with the risk of "superintelligence"?

I'm not sure I see the religious aspect.

It's more of an inevitable consequence (over an unknown timeline, granted) of the assumption that human intelligence is not some kind of absolute limit, and can be reproduced in a non-biological system. I admit that is still an assumption, but the opposite seems harder to support to me, though we're all admittedly guessing here.

Once you make that assumption, then yes of course it follows that the 'beyond human intelligence' would be able to do all of the things that human intelligence is already doing. Namely, science, mathematics, developing AI systems etc.

But there is a clear line of reasoning? AI has to be made with some kind of goal. Most goals don’t line up with humanity’s interests, and the AI will pursue them to the extreme as it’s designed for optimise aggressively for it. AI is easier to improve than biology, so once it gets good enough it will exponentially improve itself, as that allows it to achieve its goal better. Most solutions to this problem don’t actually work, we’ve yet to find a good one. It seems extremely short sighted to never consider any future situations that aren’t currently happening but seem likely to: https://xkcd.com/2278/

There is a good real world example of how optimising for a goal aggressively can make the world unliveable for some species, which is humans destroying the environment for our economic growth. But at least there we do care about it and are trying to change it, because, among other reasons, we need the environment to survive ourselves. An AI wouldn’t necessarily need that, so it would treat us more like we treat bacteria or something, only keeping us around if necessary or if we don’t get in the way (and we will get in the way)

It might sound silly but there is some solid reasoning behind it

Let me rephrase: a clear line of reasoning that isn't an Old Testament god with some Tron lines.

Also, life isn't fair, there's nothing that says we only get a manageable amount of problems at once.

It's Pascal's Wager, right (https://plato.stanford.edu/entries/pascal-wager/)?

Sure, the probability of AI-triggered extinction is lowish, but the consequences are essentially infinite, so we are justified in focusing entirely on the AI-extinction threat. (But I guess it's not infinite enough to justify rounding up all the AI researchers and shooting them?)

And just like Pascal's Wager, it's fucking bollocks. You can't just make up numbers and probabilities and imagined scenarios to invent the outcome you want.

It's rhetorical sleight of hand.