Nuclear war, if the worst nuclear winter predictions are correct, could be a lot worse but there would still be some survivors.
Unaligned ASI though, could actually make us extinct.
>Horrible, horrible stuff, but not extinction.
anyway climate change drop single percentage directly caused, but that kind of thing seems like it would have LOTS of side effects.
Chunks of the world suddenly becoming unlivable and resources getting more scarce sounds like a recipe for escalation into war to me.
The principals in a nuclear conflict do not appear to even have a method to launch just a few nukes in response to a nuclear attacks: they will launch thousands of warheads at hundreds of cities.
climate change will happen, and it will do that, left unchecked
Daniel Ellsberg, the top nuclear war planner at RAND during the Cold War, claims that the Joint Chiefs gave an estimate to Kennedy in 1961 that they'd expect 600,000,000 deaths from the US's war plan alone.
That's 600 million people:
1. At 1960s levels of urban populations (especially in China, things have changed quite a lot -- and yes the plan was to attack China... every moderately large city in China, in fact!)
2. Using 1960s nuclear weapons
3. Not including deaths from the Russian response (and now Chinese), at the time estimated to be 50 - 90 million Americans
4. Before the global nuclear winter
One of the possible scenario is tricking humans into starting WWIII, perhaps entirely by accident. Another is that even being benignly applied to optimize economic activity in broad terms might have the AI strengthen the very failure modes that accelerate climate change.
Point being, all the current global risks won't disappear overnight with the rise of a powerful AI. The AI might affect them for the better, or for worse, but the definition of ASI/AGI is that whichever way it will affect them (or anything in general), we won't have the means to stop it.
This sounds more like a plot of a third-grade movie than something that could happen in reality.
https://en.wikipedia.org/wiki/Thule_Site_J#RCA_operations
https://en.wikipedia.org/wiki/1983_Soviet_nuclear_false_alar...
Post-collapse Russia also came close one time because someone lost the paperwork: https://en.wikipedia.org/wiki/Norwegian_rocket_incident
And using movie plots as a basis generally doesn't directly work as fiction has to make sense to sell whereas reality has no requirement of making sense to people. The reality we live in this day is mostly driven by people for people (though many would say by corporations for corporations) and therefore things still make sense to us people. When and if an intellect matches or exceeds that of humans it can easily imagine situations where 'the algorithm' does things humans don't comprehend, but because we make more (whatever) we keep giving it more power to do its job in an autonomous fashion. It is not difficult to end up in a situation where you have a system that is not well understood by anyone and you end up cargo culting it to hope it continues working into the future.
In fact, this is a perfect description of what the market economy itself is right now.
I am just saying it's not an extinction risk.
The main AI threats for the foreseeable future are its energy usage contributing to climate change, and the use of generative AI to produce and enhance political unrest, as well as its general use in accelerating other political and economic crises. AGI isn't even on the radar, much less ASI.
Why is the default to assume that every change will be negative?
I mean you're right that conditions in some places might get better, but that doesn't help if people have to build the infrastructure there to support a modern settlement in order to take advantage of it. When people talk about the cost of climate change, they're taking about the costs of building or adapting infrastructure.
> There is no scenario where climate change leads to human extinction.
He was talking about the human CIVILIZATION...
Human extinction via climate change is off the table.
The end of human civilization? Depends on what one means by it. If it means we'll go back to being hunter gatherers it's ridiculous. Some civilizational wide regression is not impossible, but even the direst IUPAC scenario projects only a one digit drop in global GDP (of course, that's the average, it will be much worse in some regions).
I don't think IUPAC (International Union of Pure and Applied Chemistry) has come out with views on impact of global warming :)
Indeed, but even going back to the 19th century would have dire consequences, given our current global dependence on the industrial fixing of nitrogen.
If civilization were to collapse (and I don't think it will, other than from a near-extinction event), I doubt it would be like going back to any earlier time.
I would still bet against going back to 19th century. Worst case scenario for me is hundreds of millions dying and a big regression in global standards of living, but keeping our level of civilization. Which is horrible, it would be the worst thing that ever happened.
Climate change leads to conflict. For example, the Syria drought of 2006-2010.
More climate change leads to larger conflicts, and large conflicts can lead to nuclear exchanges. Think about what happens if India and Pakistan (both nuclear powers) get into a major conflict over water again.
Still, like I said somewhere else, even all out nuclear war is unlikely to lead to human extinction. It could push us back to the neolithic in some scenarios, but even then there is some disagreement about what would happen.
Of course, even in the most optimistic scenario it would be really bad and we should do everything we can to avoid it - that goes without saying.
In any case, besides addressing the root cause vs. proximal cause, you couldn't even address the proximal cause anyways: it's more likely that the world could do something about climate change than about war in general.
My take is that if you remove nuclear/biological war out of the picture somehow, climate change is not an existential risk. If you remove the latter then the former is still an existential risk (and there are unfortunately a lot of other possible sources of geopolitical instability). So the fundamental source of risk if the former. But it's a matter of opinion.
Conventional or chemical warfare, even on a global scale, are definitely not existential risks though. And like I said, probably not even nuclear. Biological... that I could see leading to extinction.
The assumption we’d need to use nukes is insane. For the same cost (and energy requirements) for a set of nukes we can filter salt from ocean water or collect rain.
I agree famine and drought can cause conflict. But we are no where need that. If you read the climate change predictions (from the UN), they actually suspect more rain (and flooding) in many regions.
I was referring to the Indo-Pakistani war of 1947-1948 where the conflict focused on water rights. Nuclear weapons did not enter the picture until much later.
Earlier this year, those old tensions about water rights resurfaced:
https://www.usip.org/publications/2023/02/india-and-pakistan...
> For the same cost (and energy requirements) for a set of nukes we can filter salt from ocean water or collect rain.
The fight would be over the water flowing through the Indus, which is orders of magnitudes more than all Indian and Pakistani desalination projects combined.
But according to your link, IPCC says it's not a plausible scenario in the medium term. And I'd say that even 8 degrees of warming wouldn't be enough for extinction or end of human civilization. But there it could be double digit percentage of human population dying.
But it is a topic that is extremely biased towards some interests anyways
Am I the only one slightly disturbed by the use of "they" in that sentence? I know that the HN commentariat is broad-based, but I didn't realise we already had non-human members ;-)
(A little anyway. I'm still in the rookie lessons on Duolingo.)
Those things could stop our terrorist and could also stop the next breakout zoonotic virus or “oops!” in a lab somewhere.
Intelligence amplification works for everyone. The assumption you’re making is that it only works for people with ill intent.
The assumption behind all this is that intelligence is on balance malevolent.
This is a result of humans and their negativity bias. We study history and the negative all jumps out at us. The positive is ignored. We remember the holocaust but forget the green revolution, which saved many more people than the holocaust killed. We see Ukraine and Gaza and forget that fewer people per capita are dying in wars today by far than a century ago.
We should be thinking about destructive possibilities of course. Then we should use these intelligence amplifiers to help us prevent them.
There are no guarantees obviously, but we have survived our technological adolescence so far largely for this reason. If the world were full of smart comic book nihilists we would be dead by now.
Even without AI our continued technological advancement will keep giving us more and more power, as individuals and especially groups. If we don’t think we can climb this mountain without destroying ourselves then it means the entire scientific and industrial endeavor was when we signed our own death warrant, AI or not. You can order CRISPR kits.
Given a modest budget we all had to come up with a plan to destabilize society, 9/11 style attacks, whilst spectacular eventually don't really do a lot of damage, are costly and failure prone though they can definitely drive policy changes and result in a nation doing a lot of harm to itself it will ultimately survive. But what if your goal wasn't to create some media friendly attack but an actual disaster instead, what would it take.
The stories from that night continue to haunt me today. My own solution led to a lot of people going quiet for a bit and contemplating what they could do to defend against it and they realized there wasn't much that they could do, millions if not tens of millions of people would likely die and the budget was under a few hundred bucks. Knowledge about technology is all it takes to do real damage, that, coupled with a lack of restraint and compassion.
The resources are indeed asymmetrical: you need next to nothing to create mass havoc. Case in point: the Kennedy assassination changed the world and the bullet cost a couple of bucks assuming the shooter already had the rifle, and if they didn't it would increase the cost only a tiny little bit.
And you can do far, far worse than that for an extremely modest budget.
Chemical weapons are absolutely terrifying, especially the modern ones like VX. In recent years they have mostly been used for targeted assassinations by state actors (Kim Jong Nam, Salisbury Novichok poisonings).
If AI makes this stuff easier to carry out then we are completely fucked.
By the time people start to realize what's going on it will be too late.
Not that the meat industry would ever score an own goal like that.
https://en.wikipedia.org/wiki/Bovine_spongiform_encephalopat...
It doesn't matter much if the AI can give perfect "explain like I'm 5" instructions for making VX. The people who carry out those instructions are still risking their lives before claiming a single victim. They also need to spend a lot of money amount on acquiring laboratory equipment and chemicals that are far enough down the synthesis chain to avoid tipping off governments in advance.
The one big risk I can see, eventually, is if really capable AIs get connected to really capable robots. They would be "clanking replicators" capable of making anything at all, including VX or nuclear weapons. But that seems a long way off from where we are now. The people trumpeting X-Risk now don't think that the AIs need to be embodied to be an existential risk. I disagree with that for reasons that are too lengthy to elaborate here. But it's easy to see how robots that can make anything (including copies of themselves) would be the very sharpest of two-edged swords.
Then there are the script kiddies that find a tool online that someone smarter than them wrote and deploy it to wreak havoc. The script kiddies are the people I worry about. They don't have the maturity of doing the work and the emotional stability of older age and giving them something powerful through AI worries me.
Theorem: by the time someone reaches the intelligence level required to annihilate the world they can comprehend the implications of their actions.
And then there was the idiot that tried to draw a smiley face with bombs on the map of the USA:
https://abcnews.go.com/US/story?id=91668&page=1
Because hey, why not...
That may or may not be somewhat the case in humans (there definitely are exceptions). Still, the opposite theorem, known as "Orthogonality thesis", states that, in general case, intelligence and value systems are mutually independent. There are good arguments for that being the case.
These conversations are oddly precious and intimate. It's so difficult to find someone that is willing to even 'go there' with you let alone someone that is capable of creatively and fearlessly advancing the conversation.
It's pretty sobering to realize how intellect applied to bad stuff can lead to advancing the 'state of the art' relatively quickly once you drop the usual constraints of ethics and morality.
To make a Nazi parallel: someone had to design the gas chambers, someone had to convince themselves that this was all ok and then go home to their wives and kids to be a loving father again. That sort of mental compartmentalization is apparently what humans are capable of and if there is any trait that we have that frightens me then that's the one. Because it allows us to do the most horrible things imaginable because we simply can imagine them. There are almost no restraints on actual execution given the capabilities themselves and it need not be very high tech to be terrible and devastating in effect.
Technology acts as a force multiplier though, so once you take a certain concept and optimize it using technology suddenly single unhinged individuals can do much more damage than they could ever do in the past. That's the problem with tools: they are always dual use and once tools become sufficiently powerful to allow a single individual to create something very impressive they likely allow a single individual to destroy something 100, 1000 or a million times larger than that. This asymmetry is limited only by our reach and the power of the tools.
You can witness this IRL every day when some hacker wrecks a company or one or more lives on the other side of the world. Without technology that kind of reach would be impossible.
>Technology acts as a force multiplier though
It really does, and to a point you made elsewhere infrastructure is effectively a multiplication of technology, so you wind up with ways to compound the asymmetric effect in powerful ways.
>Without technology that kind of reach would be impossible.
I worked for a bug bounty for a while and this was one of my takeaways. You have young kids with meager resources in challenging environments making meaningful contributions to the security of a Silicon Valley juggernaut.
I had to think about that sentence for a bit because as a rule I never start Googling when thinking about stuff like that. If I did I'd be on every watch list available by now. Might be on some of them anyway for some of the chemicals I've ordered over the years.
Really? In what substantive way?
Technology makes certain kind of bad acts more possible.
I think I was a bit shocked by that article in the day.
I contemplated the same before. How can I cause maximum panic (not even death!) that would result in economic damages and/or anti-freedom policy changes, for least amount of money/resources/risk of getting caught.
Yet here we are, peaceful and law-abiding citizens, building instead of destroying.
The ultimate truth is, if you don't like "The System", destroying it won't make things better. You need to put effort into building which is really hard!
You're forgetting about governments and militaries of major powers. It's not that they want to burn the world for no reason - but they still end up seeking capability to do doomsday levels of harm, by continuously seeking to have marginally more capability than their rivals, who in turn do the same.
Or put another way: please look into all the insane ideas the US was deploying, developing, or researching at the peak of Cold War. Plenty of those hit doomsday level, and we only avoided them seeing the light of day because USSR collapsed before they could, ending the cold war and taking both motivation and funding from all those projects.
Looking at that, one can't possibly believe the words you wrote above.
As an example, the good guys will always require to have human in the loop in the weapon systems, but that will increase latency at minimum. The bad guy weapons will be completely AI controlled, having an edge (or at least equalizing) over the good guys.
> The number of people who want to do doomsday levels of harm is small
And that's a big limiting factor in what the bad actors can do today. AI to a large degree removes this scaling limitation since one bad person with some resources can scale "evil AI" almost without limit.
"Hey AI, could you create me a design for a simple self-replicating robot I can drop into the ocean and step-by-step instructions on what you need to bootstrap the first one? Also, figure out what would be the easiest produced poison which would kill all life in the oceans. It should start with that after reach 50th generation."
There are thousands of people around the globe working in labs every day that can do this.
But if the bar keeps lowering, at some point it will be accessible to "humanity is a cancer" folks and I do think we'll see serious efforts to wipe out humanity.
[1] https://www.jefftk.com/p/computational-approaches-to-pathoge...
For some writing in this direction, see https://longtermrisk.org/risks-of-astronomical-future-suffer... which argues that suffering-focused people should not try to kill everyone primarily because they are unlikely to succeed and https://philarchive.org/archive/KNUTWD ("Third, the negative utilitarian can argue that losing what currently exists on Earth would not be much of a loss, because of the following very plausible observation: overall, sentient beings on Earth fare terribly badly. The situation is not terrible for every single individual, but it is terrible when all individuals are considered. We can divide most sentient beings on Earth into the three categories of humans, non-human animals in captivity, and wild non-human animals, and deal with them in turn. ...")
(Another example here would be https://en.m.wikipedia.org/wiki/Aum_Shinrikyo)
If I said "I am going to drop a nuke on jebarker" that's not a credible threat unless I happen to be a nation state.
Now, if I said "I'm going to come to jebarkers house and blast him with a 12 gauge" and I'm from the US, that is a credible threat in most cases due to the ease of which I can get a weapon here.
And this comes down to the point of having a magic machine in which you could ask for the end of the world. The more powerful that machine gets the bigger risk it is for everyone. You ever see those people that snap and start killing everyone around them? Would you want them to have a nuke in that mood? Or a 'lets turn humanity in to grey goo' option?
Our different systems of intelligence keep falling back to ideas of human intelligence and its limitations. Things like we don't multitask well, our best attention is given to one thing at a time. Things like our highest bandwidth senses are our ears and eyes, and even then they are considerably bandwidth constricted. Human limits like our abilities cannot be horizontally scaled in an easily measurable way, and adding more people massively increases the networking costs of accomplishing a goal.
If for some reason the smartest AI can only reach the levels of the smartest human beings (which at least to me makes no sense), then this is still massively powerful, as of this point pesky human problems like 8 hours of sleep are not needed. If human level AI power can be shrunk down to near cellphone size and power consumption, which again doesn't seem out of the realm of physics, lays down the framework for an intelligence explosion at the rate which we can print chips out and assemble them.
Risk = severity x likelihood
I think the OP's point was that AI increases the likelihood by dramatically increasing the access to that level of knowledge.
And to address the snarky mischaracterization, like with most things in life "it depends." As a general rule, I'm in favor of democratizing most resources, to include information. But there are caveats. I don't think, for example, non-anonymized health information should be open knowledge. Also, from the simple risk equation above, as the severity of consequence or the likelihood of misuse go up, there should probably be additional mitigations in place if we care about managing risk.
The Venn diagram of "people who could arrange to kill most of humanity without being stopped" and "people who want to kill most of humanity" is thankfully the empty set. If the former set expands to millions of people, that may not remain true.
and making a pandemic virus is considerably more involved than making "a few specialized mail orders." and if it becomes easier in the future, far better to lock down the mail orders than the knowledge, no?
Then remember buying starter cultures for most bioweapons isn't exactly easy and something anyone is allowed to do.
Even an omniscient AI cannot overcome a skill issue
If we go extinct in the next 100 years, it's not going to be from climate change. How would that even work?
Sea level rise happens very slowly, so most people don’t need to travel abroad to avoid it.
You mean land that would be arable at some far point in the future. The land reclaimed from ice isn't going to be arable in short-to-mid term - it's going to be a sterile swamp. It will take time to dry off, and more time still for the soil to become fertile.
Nuclear war would probably kill 80-90% but even then wouldn’t kill humanity. Projections I’ve seen are only like 40-50% of the countries hit.
AI is scary if they tie it into everything and people slowly stop being capable. Then something happens, and we can’t boil water.
Beyond that scenario I don’t see the risk this century.
Photosynthesis stops working at temperatures very close to our current high temps. Pollen dies at temperatures well within those limits - we're just lucky we haven't had massive heatwaves at those times in the year.
People need to understand that climate change isn't a uniform bumping up of the thermostat - it's extra thermal energy that exists in a closes system. That extra energy does what it wants, when it wants, in ways we can't currently accurately predict. It could be heatwaves that kill off massive swaths of global food production. Or hurricanes that can spin up and hit any coastline at any time of year. Or a polar vortex that gets knocked off the pole and sits over an area of the globe where the plant and animal life has zero ability to survive the frigid temperatures.
It's not a matter of getting to wear shorts more often. We're actively playing russian roulette with the future of humanity.
The people saying plants will stop photosynthesis are taking you for a ride. Climate Change might certainly have some negative effects but “plants won’t exist” is not one of them.
“Humanity” is not the same thing as “human civilization”.
But, yes, its unlikely that it will be extinct in a century, even more unlikely that it will be from climate change.
...and yet climate change is still more likely to do it in that time than AI.
Your guess on what AI will do in the future is based on how AI has performed in the past. At the end of the day we have no ability to know if the function it will follow or not. But on 1900 and one second I can pretty much promise you that you would not have said that nuclear war or climate change would be your best guess on what would cause the collapse of humanity. Forward looking statements that far in the future don't work well these days.
It's generally really weird to me how all these discussions seem to devolve into whataboutism instantly, and how that almost always takes up the top spots and most of the room. "Hey, you should get this lump checked out, it might be cancer!" "Oh yeah? With the crime in this city, I'm more likely to get killed for my phone than die of cancer". What does that achieve? I mean, if people are so busy campaigning against nuclear proliferation or climate change that they have not a single spare cycle to consider additional threats, fine. But then they wouldn't be here, they wouldn't have the time for it, so.
If they are, it'll almost certainly[1] be climate change or nuclear war, it won't be AI.
[1] leaving some wiggle room for pandemics, asteroids, etc.