They don't. They hardly talk about anything else except AI. From the outside it looks a lot like an obsession.
Personally I think they treat AI as an existential threat above all others because they consider themselves to be the pinnacle of intelligence in the universe and AI is showing that human intelligence might not be so special after all. So it's an existential threat to their sense of self. Other existential threats merely threaten their lives and are therefore not as important.
This would be fine, if acknowledged with a degree of self awareness. But it's not, instead it's wrapped up in so many layers of convoluted logic that most people, insiders and outsiders both, can't see the real reason for this obsession.
https://www.effectivealtruism.org/articles/introduction-to-e...
https://www.lesswrong.com/posts/JmmA2Mf5GrY9D6nQD/four-focus...
https://www.givingwhatwecan.org/best-charities-to-donate-to-...
That's really not true. There's a huge overlap in these groups, but the top EA charities are not shoveling all their money at AI alignment orgs... even if one can argue that they should.
How many EA people do you know?
That seems to get the most clicks, but I usually associate them with things like mosquito bed nets to help prevent malaria, because that's a far more common topic in my experience.
I'm sure you can find particular people who are all about the AI risk stuff that seems pretty wild to me, too.
But when you say "focused exclusively" I feel like you may have mostly seen news articles about wild AI stuff and not the average EA enthusiast who spends their time talking about picking efficient charities.
That isn't to say that the malaria stuff isn't still a big topic, but there has been a shift.
Longtermist topics are the perfect combination of high-prestige and smart sounding with a complete lack of accountability, because the impact of research or projects won't materialize for decades. Ironically, in practice it directly contradicts the founding tenants of EA, despite being a logical conclusion of the underlying moral philosophy.
Just looking at the topics section of the EA forums (https://forum.effectivealtruism.org/topics/all), we've got
- 466 posts in AI safety
- 1060 posts in existential risk (seems like a roughly even mix of pandemics, nuclear war, and AI)
- 816 posts in global health
- 193 posts in animal welfare
- 42 posts in biosecurity & pandemics
It would be interesting to understand how much of the "existential risk" category is devoted to biosecurity or pandemics, and how recent those posts are.
Pending the above, my tentative takeaway from this is that COVID - arguably the greatest threat to human health and wellbeing in the past several decades - seems to have been a minor/non-existent blip on their radar until it actually happened. This raises serious questions in my mind about the community's predictive abilities and/or bias.
This is a very low bar to jump over!
Anyway I read through that article. There's a lot of guff about Newcomb's paradox and Eliezer, and a single paragraph referencing COVID, with two links supporting your statement. I've clicked through those links and have come out unimpressed.
The first link is used to supported the claim that "The rationalist community was well ahead of the public": https://slatestarcodex.com/2020/04/14/a-failure-but-not-of-p... .
Scott rightly highlights how the media and institutions got things wrong, but conversely gives only a few examples of "generic smart people on Twitter" getting some things right in the early days of the outbreak. He confesses that he himself did not predict the seriousness of COVID.
The second link is used to support the claim "the wider tech community, were using masks and stocking up on essential goods, even as others were saying to worry about the flu instead": https://putanumonit.com/2020/02/27/seeing-the-smoke/ .
The linked article was written at the end of February, when panic-buying had already firmly set in across the US.
There is nothing here about the reasoning or predictive abilities of the rationalist or EA community specifically. Nor is there any compelling comparison of its response compared to the wider public.
It looks like the Forum is confusing here: it has "Biosecurity & Pandemic Preparedness" [1] as one topic, and "Biosecurity" [2] as its largest subtopic. But when you click on the parent topic it's not showing the subtopic posts; I've filed a bug.
Older good biosecurity posts include:
* 2014: Good policy ideas that won’t happen (yet): https://forum.effectivealtruism.org/posts/n5CNeo9jxDsCit9dj/...
* 2017: Biosecurity as an EA Cause Area: https://forum.effectivealtruism.org/posts/eXgCREwL5PQWHEtvm/...
* 2021: Concrete Biosecurity Projects (some of which could be big): https://forum.effectivealtruism.org/posts/u5JesqQ3jdLENXBtB/...
[1] https://forum.effectivealtruism.org/topics/biosecurity-and-p...
"Some of the reasons I'm skeptical of natural risks are that first of all, they've never really happened before. Humans have obviously never been caused to go extinct by a natural risk, otherwise we would not be here talking. It doesn't seem like human civilization has come close to the brink of collapse because of a natural risk, especially in the recent past.
"You can argue about some things like the Black Death, which certainly caused very severe effects on civilization in certain areas in the past. But this implies a fairly low base rate. We should think in any given decade, there's a relatively low chance of some disease just emerging that could have such a devastating impact. Similarly, it seems like it rarely happens with nonhuman animals that a pathogen emerges that causes them to go extinct. I know there's one confirmed case in mammals. I don't know of any others. This scarcity of cases also implies that this isn't something that happens very frequently, so in any given decade, we should probably start with a prior that there's a low probability of a catastrophically bad natural pathogen occurring."
I wonder if this is a case of extrapolation fallacy? Modern human dynamical behaviour is significantly different to both animal behaviour and less modern behaviour. Viruses spread more easily in the age of global travel; those that used to be too deadly to spread very far suddenly have more opportunity.
EDIT: Reading this more carefully, the speaker does actually address globalisation, but seems to dismiss it as a counterargument and I'm not really sure why.
I know I'm jumping the gun just a little, but c'mon. An apocalypse list without climate?
The AIpocalypse seems incredibly unlikely. I'm a lot more worried about the nukes, and even if we end up with robotic overlords, I'd bet they'll be a whole lot better at administration than meat -admins have proven.
I'm with you on engineered superviruses. Feasible, likely, and incredibly high impact.
What I kind of keep coming back to is the risk profile of all this stuff. That magic vector of likelihood * impact. Global warming is happening now. And it's real bad - worse than I think we give it credit for.
What I worry about is that we're the proverbial frog in the pot. Things get just slightly hotter each year, so we'll miss it when we actually boil.
1 https://www.google.com/search?q=double+asteroid+redirection+...
My comment was written with the summary of that article in mind (I didn't make this clear):
> Climate change is going to significantly and negatively impact the world. Its impacts on the poorest people in our society and our planet’s biodiversity are cause for particular concern. Looking at the worst possible scenarios, it could be an important factor that increases existential threats from other sources, like great power conflicts, nuclear war, or pandemics. But because the worst potential consequences seem to run through those other sources, and these other risks seem larger and more neglected, we think most readers can have a greater impact in expectation working directly on one of these other risks.
There is an excellent chapter on the existential risks associated with climate change in Toby Ord's book The Precipice, which you can get a free copy of at https://80000hours.org/the-precipice/
The topics listing isn't great, but there's been a lot of discussion about climate change on the forum: https://forum.effectivealtruism.org/topics/climate-change (239 posts)
The way the EA forum handles parent topics is confusing, but this is only counting posts tagged with the top-level label. It has several subcategories, and there are hundreds of posts when you include them:
* 218: Biosecurity
* 176: Covid
* 54: Pandemic preparedness
* 40: Global catastrophic biological risk
* 39: Vaccines
* 20: Biotech
* 16: Life sciences
* 11: Dual-use
* 10: Biosurveillance
(Posts can have multiple labels, so the above list double-counts a bit. I don't see an easy way to extract the actual count from the forum, but it's at least 218, the count for the largest category.)
Even if that were true, there are lots of people already looking at those other existential risks. The EA crowd was the only group concerned about AI for a long, long time. Seeing an existential risk that nobody else does leads to evangelism and hyper focus so people start paying attention.
But since the real goals of the EAs are not to do any of that (it's just a bunch of socially-awkward geeks looking to feel accepted in some kind of community and find sexual partners; in other words, trying to feel like normal humans), they fooled themselves into believing that such short term actions are "ineffective"; actually, any actual measurable hard work has to necessarily be ineffective, or else it would actually have to be done. So they keep pushing their agenda to longer and longer term issues, more abstract and less well-defined as possible, which cannot be actually measured, so they can continue to build this weird, meaningless, ridiculous fantasy that they live in (and which supports their social hierarchy) without having to actually do anything concrete.
Any reason to believe this? The human brain runs on 20 watts (!). Obviously biological hardware is not directly comparable to silicon, perhaps silicon is intrinsically less efficient by some factor such that in order to obtain 20W of human brain compute we need to expend K times more in silicon. But is that factor really so large that we are compute-bound? How big would K have to be for us to be compute-bound anyway? If you could have Einstein for 200kW (i.e. silicon is intrinsically 1000x less efficient than brains) would that be compute-bound? It seems much more likely that we are algorithm-bound and simply not using our available compute efficiently.
Yudkowsky's hard take off mentions "solving protein folding", nanomachines, persuading everyone to let them connect to the internet, infinite adaptability, etc. I think those are unrealistic, but obviously they still have a non-zero chance.
More probably bad possibilities are AI-Hitler (monomaniacal populist absolute leader) or AI-Stalin (manipulative, smart, absolutely paranoid, rising through the ranks) ... so something that's human-like enough to be able to connect with humans, to manipulate them, but at the same time less affected by the psychological shortcomings. (Ie. such an AI could spend enough time to cross-interrogate every underling, constantly watch them, etc.)
And yes, a very efficient immortal dictator is very bad news, but still bound by human-like limits.
And the big infinite dollar question is could this hypothetical AI improve on itself by transcending human limits? Let's say by directly writing programs that it has conscious control over? Can it truly "watch" a 1000 video streams in real-time?
Can it increase the number of its input-and-output channels while maintaining its human-like efficiency?
Because it's very different to run a fast neural network that spits out a myriad labels for every frame of a video stream (YOLO does this already, but it's not 20W!) and to integrate those labels into actions based on a constantly evolving strategy.
Sure maybe the hypothetical AI will simply run a lot of AlphaZero-like hybrid tree-search estimator-evaluator things ...
Anyway, what I'm trying to say is that our 20W efficiency comes with getting tried very fast, and using "fast mode" thinking for everything. (Slow mode is the exception, and using it is so rare that we basically pop open a glass of champagne every time.)
>And the big infinite dollar question is could this hypothetical AI improve on itself by transcending human limits? Let's say by directly writing programs that it has conscious control over? Can it truly "watch" a 1000 video streams in real-time?
Even if its mind wasn't truly directly scalable, it could make 1000 short or long-lived copies of itself to delegate those tasks to.
What’s the upper bound for climate change’s existential risks? The end of human existence and society as we know it except for a relatively small number of survivors living in a world those of us alive today can barely imagine?
https://www.scientificamerican.com/article/impact-from-the-d...
When the article was written, CO2 concentration was 385 ppm. Today it's 421 ppm. If CO2 concentrations were to rise linearly, the timeline for reaching 1,000 ppm would be 250 years. Achieving that would require emissions to stabilize at ~2014 levels. If emissions keep rising, then perhaps it's closer to 150 years. If strong feedbacks kick in, like methane from melting permafrost, then maybe 100 years or less, or maybe we'll reach that happy 2500 ppm mark and see what a real mass-extinction looks like.
Of course the ocean has a big thermal mass, so it will probably take quite some time to heat up enough to trigger the disaster even after we reach that level of atmospheric carbon. Hopefully everything will be fine?
It matters a lot. Whilst I was eating dinner I've now read the article you cited, the paper that it cites (Kump, Arthur & Pavlov, 2015). I also cross checked the argument and numbers against several other papers. Ward's argument is extremely slippery. This is sadly what I'm coming to expect from academics with unverifiable models. You have to read everything they write adversarially.
"In [Kump2015]'s models, if the deepwater H2S concentrations were to increase beyond a critical threshold during such an interval of oceanic anoxia, then the chemocline separating the H2S-rich deepwater from oxygenated surface water could have floated up to the top abruptly. The horrific result would be great bubbles of toxic H2S gas erupting into the atmosphere."
Observe that his argument starts with a "critical threshold" for H2S levels, not CO2, and that he doesn't tell us what this critical level is. The obvious questions are thus: what is this level, is it realistic for our present day oceans to reach this critical level and if so, how? To get to those levels of H2S he hypothesises that:
"if ancient volcanism raised CO2 and lowered the amount of oxygen in the atmosphere, and global warming made it more difficult for the remaining oxygen to penetrate the oceans, conditions would have become amenable for the deep-sea anaerobic bacteria to generate massive upwellings of H2S."
In other words in this theory the supervolcanoes have to come first, triggering a sharp drop in oxygen levels in the atmosphere, which in turn then causes the chemocline to move, which then causes the H2S upswellings. It's all caused by a massive loss of oxygen, not increase in CO2 levels, which is simply another result of the volcanisms. Kump2015 also makes it clear that their hypothesis requires a truly massive drop in oxygen levels to occur. We'll look at how much in a moment.
But just a few paragraphs later Ward has forgotten all about the volcanoes and oxygen levels. Suddenly it's all about absolute CO2 levels - not even rates! Cause, effect and unrelated side effects have become entirely muddled, probably because he knows nobody would care about his article unless he ties it to global warming armageddon somehow and because hey, this entire field is nothing but assumptions, suppositions, and playing with numbers that can never be verified anyway so why not?
His argument is brittle in other ways. He asserts without any backing argument that whilst supervolcanoes explain all the other non-asteroid mass extinction events it doesn't for the Permian because, apparently, supervolcanoes are really great for plants on land which can "probably" survive the warming. And there was me thinking that large scale volcanic activity is supposed to be very bad for plants because it dims the atmosphere:
https://www.vulkane.net/en/volcanism/supervolcano.html#:~:te....
"Plant growth is restricted and mass extinction can be caused."
Back to the H2S claims. Turning our attention to the Kump2015 paper we find immediately that it's got an annoying structure in which they work backwards from their desired scenario to calculate the level of H2S that could trigger it:
"Thus, if the H2S of the deep sea increased during an anoxic interval beyond a critical value (1 mmol/kg), upwelling regions of the world ocean would become sulfidic, even with the modern [oxygen level at atmospheric pressure] .... A slightly more sulfidic ocean with H2S = 3 mmol/kg ... could sustain ... 2000 times the present-day flux and a critical value for the atmosphere (see following)."
But these critical values are never placed clearly in context. Is 3 millimols a lot or not much? Note that the 3 mmol/kg value is the absolute "best" case for their scenario; it can only be that low in (they estimate) 0.1% of the world's oceans because normal ocean requires levels 20x higher. They do admit that:
"The [H2S] condition is extreme, and thus likely to have been rarely achieved in Earth history. Is there any evidence that such conditions have occurred in the geologic past?"
How extreme is it? The value of the normal ocean would be important to have here but they don't give it to us. The paper "Hydrogen Sulfide in the Black Sea" by Volkov & Neretin does give values though, and bear in mind this is by far the most anoxic basin in the world: at a depth of 1km the value is ~314 micromols/kg. So these H2S levels that are claimed to trigger this process are literally many, many orders of magnitude higher than even the Black Sea.
The levels of oxygen drop needed is something they also don't seem to directly share, but in one section they're kicking around a figure like half of all today's oxygen having vanished, or for a different event, 99% of all oxygen having gone.
Even putting aside that these papers are just piles of completely unverifiable suppositions stacked like a jenga tower, there is absolutely nothing even remotely close to realistic about this scenario happening to us. It requires oxygen levels to drop so much that if it were to ever become an actual threat we'd all have died of oxygen starvation long before.
Still, I certainly do hope that you're correct!
Except at most one.
AI is...well, people losing their shit over ChatGPT (**) aside, AI is not going to be real enough to worry about for a few more decades at least.
(*) Anyone who's about to regurgitate some fossil-fuel industry talking points in response, just save your breath.
(**) Above note goes double for this.
No, I would rather we wait until we are close enough to AI that we are talking about something concrete, rather than making wild speculations.
Would you have opposed research into renewable energy in the 1970s since global warming was still a few decades away from being something we needed to worry about?
If you think this than you have misunderstood me. Now go elsewhere and bother other people.
What makes you sure it's not going to be a problem?
Your intent is irrelevant.
And since the world exists every group will be wrong except one.
I think the risk of human extermination is much lower than many in this group believe because I think AI is much more compute bound than algorithm bound. So I think a hard takeoff is very unlikely. But while the median case for climate change is probably worse than the median case for AI. The existential risks for AI seem much higher. And it's a concern worth taking seriously.