I am old enough to remember the Cold War and the quite plausible risk that civilization would be wiped out by nuclear war.

This is similar except the people warning about nuclear weapons are eagerly soliciting venture capital to build more of them.

I concur. Climate change and ecological collapse are thing we should be focusing on. And I’m hopeful that AI in the long term might help us towards solving them. But to be honest, most normal people are just worried and scared about their jobs/financial impact which are their most immediate concerns and that’s what got most people in a frenzy. The AI gonna destroy world crowd is a bit of an outlier imo.

Climate change is not an intelligent adversary. AI tech might give rise to one.

Climate change was killing hundreds yesterday, is killing tens of thousands today, and if the trend continues, will kill millions tomorrow. AI made like 5 people kill themselves today. Pardon me if I don't pay attention to the mellow threat and want to focus on the iss that actually threaten us.

Climate change is slow, and relies on a few highly indirect mechanisms like agricultural failures and famines to kill civilization-level-significant amounts of people. This is a hard ceiling for it.

An advanced AI threat is a civilization of one. It can act, adapt and adjust rapidly, like humans do. Potentially faster than humans do. It has access to all the tools that human civilization has already invented that it can reproduce, obtain or subvert - and potentially more. It's not limited to having just one big slow lethality mechanism. It can unleash multiple civilization-scale threats in a quick succession, and keep piling them until recovery becomes impossible. Humans can adapt to a lot, but an intelligent adversary adapts back.

The threat class isn't the same, and it's not even close. Climate change is an "otherwise avoidable death and suffering" risk. AI is an actual "extinction of humankind" risk.

Are you talking about LLMs+harnesses? No. The day a LLM can play chess without harness, or make it's own harness I might listen to you, until then I'll work on actual threats. No object permanence makes any world conquering _really_ difficult.

LLMs absolutely can make their own harnesses. And tell me: where did object permanence come from in the HuggingFace incident?

Free solo climbers, who climb without harnesses, are typically regarded as having a death wish.

Yet, the dangers of AI getting smarter/capable by the month, with stronger offensive capabilities, and alignment observability and congnitive control getting worse, is "imagination".

Interestingly, denialist positions like this don't touch the unsolved problems, and _these_ are FUD on my watch.

The problems I've mentioned are very concrete, and spoiler alert: there may be no solutions to alignment observability and its possible degradation - at least with AI companies not substantially working on it (or even worse, working against it - see recurrent neural networks).

I personally believe there is no solution to alignment. Humans at best are quasi-aligned. Take England, France, and Spain some hundreds of years ago, they'd one up each other and make some gains and take some losses, even though there were bad things happening it had it's own measure of stability.

Then you have the natives of America who had a boat show up. It was wholesale genocide, mostly to disease, but a lot of murder and war. We really have no idea how many died because most died long before we came to shoot them with guns from disease. Any that resisted were brutally put down with technology.

AGI/ASI will be much more akin to a colonizer and we will be the natives in this future.

There are a couple of solutions from prominent experts:

- Yampolskiy proposes superintelligent but extremely narrow AIs (for example, one that works on a single type of cancer)

- Bengio is developing a solution at LawZero

But as long as the current companies are leading, such solutions will be essentially irrelevant.

I agree that we'll be colonized, unfortunately.