Paper clip optimizers always need a secondary (and often dubious) physical element to pose an actual threat, like grey goo or nanomachines or strange matter.

The secondary element can be humans.

Even if it only gets as far as running a Ponzi or pyramid scheme — and there are plenty of those where the scheme itself is a small block of text that convinces those reading it to (1) share it, and (2) give money to the person who first showed them this message — still breaks economics bad enough to matter, even if you don't give any credence at all to Yudkowsky etc.

Or in a more literal sense of using humans, as each separate component of ordering the creation and delivery of a custom (non-computer) virus has been demonstrated, and the only good news is the bio-labs say no to printing the D/RNA for dangerous viruses and so do the LLMs that most noobs try to use.

Pyramid schemes are a nuisance. Covid-19 should have been a wake up call that most bio labs ought to be closely re-examined or closed down, regardless of whether an AI happens to be abusing them. The bio-lab scenario is the only one I see as being an actually plausible concern, though.

I think paper clip optimizers are already here in the form of advertising and engagement algorithms, and the physical element is us.

For however unpleasant those things may be, I wouldn't call them a Doomsday scenario. They're certainly nothing like the things you'll hear of on LessWrong.

It's too early to tell if it's doomsday. There are plenty of reasons to be alarmed about the impact of social media on mental health, increasing effectiveness of algorithmic radicalization, cracking foundations of important institutions and the potential path that follows.

A path that will include an ever-more pervasive presence and influence of the next generation of this tech.

We failed with social media, and now we're plunging headlong into territory that's arguably far more treacherous and there's very little indication that we'll learn from the last time.

So now we're just starting to come to terms with the harms that we didn't see coming, while going 1000mph with a category of tech that has a myriad of foreseeable problems to say nothing of the ones we haven't thought of.

I hope there will be an equalizing positive effect, but there seem to be good reasons to worry.

The consequences of algorithmic radicalization have yet to hit the level of the radicalization in the 1930s that happened without any social media or computers at all. I think there is a deeper cause to that then just Facebook doing slimy things.

Mass radicalization depends on mass immiseration; when most people are generally satisfied with how things are going, the cranks stay on the fringe. Smart cranks know this, and spend their time on the fringe organizing access to communications such that when people cease to be satisfied and start looking for radical alternatives, they're all ready and set to go bringing their program to the masses. Facebook has already shown itself to be great for that.

The general agreement of humans at any point in time is that now is the most messed up.

That sentiment is part of society. Not sure if it could ever be changed.

See also, plausibly:

- military industrial complexes

- questionably Democratic democracies, riddled by deception and lobbying

- a financial system that doesn't seem to be particularly constrained by the political system

- yet another major war underway, and another one on deck

- an epistemic & trust crisis

- major political polarization

- global cost of living crises

And we accomplished all of this in record time, with dumb computers!

Did I miss anything or is everything else more or less a-ok?

I agree with most of these but the cause isn’t computers. Computers are just another thing to add to the list

* * *

If you look deeply enough into the 'complex causality' you'll find anything you want. It doesn't mean much by itself.

You may, but not me.

Though, I suspect you are like most people in that if you gaze into the unknown (which is what we are dealing with here), you will see what you want to see.

You see, the actual causality is unknowable, so your mind has generated a simulation of what "the" casualty is.

This is clearly obvious, is it not?

I'll play along with your attempt at armchair psychology. What makes you special that you are immune to this effect you propose I suffer from? Are you not just seeing what you want to see? Why is your simulation of causality better than mine?

Or a genetics lab hooked up to a computer...

They all are. A quite a bit of AI there too.

> The near term risk isn't super intelligence, it's the paper-clip optimizer pushing the wrong button.

This has already happened. The paper clip optimizer was capitalism.

Just listen to some heatwave weather forecasts once in a while (as an example).

There may be more than one thing. Capitalism as an information processing system is quite slow. Computers can be faster.

“Already happened” merely means that it is not a hypothetical scenario any more.

To anyone who says we can just flip the off switch on AIs who go rogue: where exactly is the off switch to capitalism, or corporations, or global supply chains, or industrialized agriculture, or social media, or the State?

AIs aren't a new phenomenon: we've been co-evolving with them for millennia, for both good and ill. The new more powerful iterations ("vast arrays of inscrutable floating-point matrices") are nested in those previous memetic life forms (corporations in particular [0]); and once something becomes infrastructure (and/or an income stream to some subset of humanity), it becomes non-trivial, if not impossible, to turn off.

[0] https://omniorthogonal.blogspot.com/2013/02/hostile-ai-youre...