News for hackers. Stuff that matters.
Re:Silicon Valley’s Obsession with Killer Rogue AI Helps Bur... by esperent on Wednesday March 08, 2023 @02:24AM (#35066223)
The issue is that the effective altruism crowd are focused exclusively on AI safety. If they talked about all possible threats to humanity and how to solve them, from AI, to climate change, to nuclear bombs, to asteroids, to disease, they would be an interesting and important group.

They don't. They hardly talk about anything else except AI. From the outside it looks a lot like an obsession.

Personally I think they treat AI as an existential threat above all others because they consider themselves to be the pinnacle of intelligence in the universe and AI is showing that human intelligence might not be so special after all. So it's an existential threat to their sense of self. Other existential threats merely threaten their lives and are therefore not as important.

This would be fine, if acknowledged with a degree of self awareness. But it's not, instead it's wrapped up in so many layers of convoluted logic that most people, insiders and outsiders both, can't see the real reason for this obsession.

Re:Silicon Valley’s Obsession with Killer Rogue AI Helps Bur... by rhn_mk1 on Wednesday March 08, 2023 @02:41AM (#35066325)
Re:Silicon Valley’s Obsession with Killer Rogue AI Helps Bur... by mitthrowaway2 on Wednesday March 08, 2023 @02:43AM (#35066339)
I think they're worried about it because they honestly do care about existential risk. If you can come up with some more likely existential threats than AI, I'm sure they'll worry about that too.
Re:Silicon Valley’s Obsession with Killer Rogue AI Helps Bur... by yanderekko on Wednesday March 08, 2023 @02:49AM (#35066376)
>The issue is that the effective altruism crowd are focused exclusively on AI safety.

That's really not true. There's a huge overlap in these groups, but the top EA charities are not shoveling all their money at AI alignment orgs... even if one can argue that they should.

Re:Silicon Valley’s Obsession with Killer Rogue AI Helps Bur... by whatshisface on Wednesday March 08, 2023 @03:38AM (#35066686)
Arguing that AI safety is needed only demonstrates that it's altruistic, but you have a lot farther to go if you want to say funding research in this decade can be effective.
Re:Silicon Valley’s Obsession with Killer Rogue AI Helps Bur... by Natsu on Wednesday March 08, 2023 @02:51AM (#35066391)
> The issue is that the effective altruism crowd are focused exclusively on AI safety.

How many EA people do you know?

That seems to get the most clicks, but I usually associate them with things like mosquito bed nets to help prevent malaria, because that's a far more common topic in my experience.

I'm sure you can find particular people who are all about the AI risk stuff that seems pretty wild to me, too.

But when you say "focused exclusively" I feel like you may have mostly seen news articles about wild AI stuff and not the average EA enthusiast who spends their time talking about picking efficient charities.

Re:Silicon Valley’s Obsession with Killer Rogue AI Helps Bur... by aetherson on Wednesday March 08, 2023 @05:57AM (#35067526)
As someone who is not an EA, but has been peripherally aware of them for years, the bed nets stuff was their big thing until maybe three years ago, but it genuinely is the case that a lot of their energy had recently redirected to AI safety and other X-risks.

That isn't to say that the malaria stuff isn't still a big topic, but there has been a shift.

Re:Silicon Valley’s Obsession with Killer Rogue AI Helps Bur... by HDThoreaun on Wednesday March 08, 2023 @10:47AM (#35070424)
Givewell is still where most EA money flows and they are almost exclusively focused on mosquito nets and other malaria intervention.
Re:Silicon Valley’s Obsession with Killer Rogue AI Helps Bur... by bakuninsbart on Wednesday March 08, 2023 @01:14PM (#35072598)
Yep, I think the community is pretty evenly split amongst "longtermist"-focused people, and this includes all the X-Risk stuff, as well as the more classical "Global Health & Happiness" people. At this point they are actually relatively distinct groups I'd wager. One thing missing from the article is that, at least in my experience, the vast majority of the opportunists are in the longtermist camp.

Longtermist topics are the perfect combination of high-prestige and smart sounding with a complete lack of accountability, because the impact of research or projects won't materialize for decades. Ironically, in practice it directly contradicts the founding tenants of EA, despite being a logical conclusion of the underlying moral philosophy.

Re:Silicon Valley’s Obsession with Killer Rogue AI Helps Bur... by arisAlexis on Wednesday March 08, 2023 @04:38AM (#35067012)
they do. One of the most famous EA books is the precipice and it talks about all existential risks. Also most of the funding gets dispersed.
Re:Silicon Valley’s Obsession with Killer Rogue AI Helps Bur... by consilient on Wednesday March 08, 2023 @04:43AM (#35067047)
> The issue is that the effective altruism crowd are focused exclusively on AI safety, at least in what they publish online.

Just looking at the topics section of the EA forums (https://forum.effectivealtruism.org/topics/all), we've got

- 466 posts in AI safety

- 1060 posts in existential risk (seems like a roughly even mix of pandemics, nuclear war, and AI)

- 816 posts in global health

- 193 posts in animal welfare

- 42 posts in biosecurity & pandemics

Re:Silicon Valley’s Obsession with Killer Rogue AI Helps Bur... by n4r9 on Wednesday March 08, 2023 @05:41AM (#35067410)
I've never been on this website before, but am I correct in understanding that the "biosecurity" categoery is about a month old?

It would be interesting to understand how much of the "existential risk" category is devoted to biosecurity or pandemics, and how recent those posts are.

Pending the above, my tentative takeaway from this is that COVID - arguably the greatest threat to human health and wellbeing in the past several decades - seems to have been a minor/non-existent blip on their radar until it actually happened. This raises serious questions in my mind about the community's predictive abilities and/or bias.

Re:Silicon Valley’s Obsession with Killer Rogue AI Helps Bur... by AgentME on Wednesday March 08, 2023 @05:56AM (#35067518)
The rationalist and EA communities are often credited for taking Covid much more seriously sooner than most and the media did. https://unherd.com/2020/12/how-rational-have-you-been-this-y...
Re:Silicon Valley’s Obsession with Killer Rogue AI Helps Bur... by n4r9 on Wednesday March 08, 2023 @06:54AM (#35067923)
> Sooner than most and the media.

This is a very low bar to jump over!

Anyway I read through that article. There's a lot of guff about Newcomb's paradox and Eliezer, and a single paragraph referencing COVID, with two links supporting your statement. I've clicked through those links and have come out unimpressed.

The first link is used to supported the claim that "The rationalist community was well ahead of the public": https://slatestarcodex.com/2020/04/14/a-failure-but-not-of-p... .

Scott rightly highlights how the media and institutions got things wrong, but conversely gives only a few examples of "generic smart people on Twitter" getting some things right in the early days of the outbreak. He confesses that he himself did not predict the seriousness of COVID.

The second link is used to support the claim "the wider tech community, were using masks and stocking up on essential goods, even as others were saying to worry about the flu instead": https://putanumonit.com/2020/02/27/seeing-the-smoke/ .

The linked article was written at the end of February, when panic-buying had already firmly set in across the US.

There is nothing here about the reasoning or predictive abilities of the rationalist or EA community specifically. Nor is there any compelling comparison of its response compared to the wider public.

Re:Silicon Valley’s Obsession with Killer Rogue AI Helps Bur... by jefftk on Wednesday March 08, 2023 @09:48AM (#35069574)
> am I correct in understanding that the "biosecurity" category is about a month old?

It looks like the Forum is confusing here: it has "Biosecurity & Pandemic Preparedness" [1] as one topic, and "Biosecurity" [2] as its largest subtopic. But when you click on the parent topic it's not showing the subtopic posts; I've filed a bug.

Older good biosecurity posts include:

* 2014: Good policy ideas that won’t happen (yet): https://forum.effectivealtruism.org/posts/n5CNeo9jxDsCit9dj/...

* 2017: Biosecurity as an EA Cause Area: https://forum.effectivealtruism.org/posts/eXgCREwL5PQWHEtvm/...

* 2021: Concrete Biosecurity Projects (some of which could be big): https://forum.effectivealtruism.org/posts/u5JesqQ3jdLENXBtB/...

[1] https://forum.effectivealtruism.org/topics/biosecurity-and-p...

[2] https://forum.effectivealtruism.org/topics/biosecurity

Re:Silicon Valley’s Obsession with Killer Rogue AI Helps Bur... by n4r9 on Wednesday March 08, 2023 @11:31AM (#35071077)
Thanks for the reading. The 2017 talk/article is interesting in light of COVID:

"Some of the reasons I'm skeptical of natural risks are that first of all, they've never really happened before. Humans have obviously never been caused to go extinct by a natural risk, otherwise we would not be here talking. It doesn't seem like human civilization has come close to the brink of collapse because of a natural risk, especially in the recent past.

"You can argue about some things like the Black Death, which certainly caused very severe effects on civilization in certain areas in the past. But this implies a fairly low base rate. We should think in any given decade, there's a relatively low chance of some disease just emerging that could have such a devastating impact. Similarly, it seems like it rarely happens with nonhuman animals that a pathogen emerges that causes them to go extinct. I know there's one confirmed case in mammals. I don't know of any others. This scarcity of cases also implies that this isn't something that happens very frequently, so in any given decade, we should probably start with a prior that there's a low probability of a catastrophically bad natural pathogen occurring."

I wonder if this is a case of extrapolation fallacy? Modern human dynamical behaviour is significantly different to both animal behaviour and less modern behaviour. Viruses spread more easily in the age of global travel; those that used to be too deadly to spread very far suddenly have more opportunity.

EDIT: Reading this more carefully, the speaker does actually address globalisation, but seems to dismiss it as a counterargument and I'm not really sure why.

Re:Silicon Valley’s Obsession with Killer Rogue AI Helps Bur... by pksebben on Wednesday March 08, 2023 @05:39AM (#35067396)
didn't really have much of a dog in this fight, but looking at that breakdown I wonder which oil company is bankrolling them.

I know I'm jumping the gun just a little, but c'mon. An apocalypse list without climate?

Re:Silicon Valley’s Obsession with Killer Rogue AI Helps Bur... by dotsam on Wednesday March 08, 2023 @06:23AM (#35067707)
Climate change is unlikely to kill everyone on earth. Other risks, such as an engineered pandemic, asteroid impact, or an AI apocalypse have the possibility of killing ~everyone. This is not saying that climate change is not a real issue.
Re:Silicon Valley’s Obsession with Killer Rogue AI Helps Bur... by pksebben on Wednesday March 08, 2023 @06:50AM (#35067894)
Well, we're past the asteroids, at least. [1]

The AIpocalypse seems incredibly unlikely. I'm a lot more worried about the nukes, and even if we end up with robotic overlords, I'd bet they'll be a whole lot better at administration than meat -admins have proven.

I'm with you on engineered superviruses. Feasible, likely, and incredibly high impact.

What I kind of keep coming back to is the risk profile of all this stuff. That magic vector of likelihood * impact. Global warming is happening now. And it's real bad - worse than I think we give it credit for.

What I worry about is that we're the proverbial frog in the pot. Things get just slightly hotter each year, so we'll miss it when we actually boil.

1 https://www.google.com/search?q=double+asteroid+redirection+...

Re:Silicon Valley’s Obsession with Killer Rogue AI Helps Bur... by esperent on Wednesday March 08, 2023 @10:06AM (#35069845)
So what you're saying is that if a risk can totally upend our society, destroy most of our cities, make most of our farmland unusable, destabilize geopolitics in a way that's almost certain to lead to war between nuclear powers (who also have the ability to engineer deadly diseases), and massively disrupt every ecosystem on earth, and that there's enough evidence to say with almost complete certainty that this will come to pass without massive societal and political change and technological intervention... but it's unlikely to kill everyone... then it doesn't really deserve mention?
Re:Silicon Valley’s Obsession with Killer Rogue AI Helps Bur... by dotsam on Wednesday March 08, 2023 @11:25AM (#35070980)
The cascading risks you mention are certainly real and serious, and are worthy of our best and urgent efforts to solve. Effective altruists are rightly concerned about these effects e.g. https://80000hours.org/problem-profiles/climate-change/

My comment was written with the summary of that article in mind (I didn't make this clear):

> Climate change is going to significantly and negatively impact the world. Its impacts on the poorest people in our society and our planet’s biodiversity are cause for particular concern. Looking at the worst possible scenarios, it could be an important factor that increases existential threats from other sources, like great power conflicts, nuclear war, or pandemics. But because the worst potential consequences seem to run through those other sources, and these other risks seem larger and more neglected, we think most readers can have a greater impact in expectation working directly on one of these other risks.

There is an excellent chapter on the existential risks associated with climate change in Toby Ord's book The Precipice, which you can get a free copy of at https://80000hours.org/the-precipice/

Re:Silicon Valley’s Obsession with Killer Rogue AI Helps Bur... by low_tech_love on Wednesday March 08, 2023 @05:58PM (#35075965)
Says whose crystal ball?
Re:Silicon Valley’s Obsession with Killer Rogue AI Helps Bur... by jefftk on Wednesday March 08, 2023 @10:02AM (#35069793)
> An apocalypse list without climate?

The topics listing isn't great, but there's been a lot of discussion about climate change on the forum: https://forum.effectivealtruism.org/topics/climate-change (239 posts)

Re:Silicon Valley’s Obsession with Killer Rogue AI Helps Bur... by jefftk on Wednesday March 08, 2023 @09:59AM (#35069753)
> 42 posts in biosecurity & pandemics

The way the EA forum handles parent topics is confusing, but this is only counting posts tagged with the top-level label. It has several subcategories, and there are hundreds of posts when you include them:

* 218: Biosecurity

* 176: Covid

* 54: Pandemic preparedness

* 40: Global catastrophic biological risk

* 39: Vaccines

* 20: Biotech

* 16: Life sciences

* 11: Dual-use

* 10: Biosurveillance

(Posts can have multiple labels, so the above list double-counts a bit. I don't see an easy way to extract the actual count from the forum, but it's at least 218, the count for the largest category.)

Re:Silicon Valley’s Obsession with Killer Rogue AI Helps Bur... by naasking on Wednesday March 08, 2023 @11:28AM (#35071024)
> They don't. They hardly talk about anything else except AI.

Even if that were true, there are lots of people already looking at those other existential risks. The EA crowd was the only group concerned about AI for a long, long time. Seeing an existential risk that nobody else does leads to evangelism and hyper focus so people start paying attention.

Re:Silicon Valley’s Obsession with Killer Rogue AI Helps Bur... by low_tech_love on Wednesday March 08, 2023 @05:56PM (#35075936)
AI safety and "longtermism" (I feel slightly more stupid just by writing this) is an escape route for the group's realization that (a) they cannot really make a difference and (b) even if they could, it would require a lot of actual work. The person who goes out in the middle of a cold night and cooks soup for homeless people can quite easily count how many mouths he fed; sounds quite effective and quantitative to me. Or maybe the volunteer who is helping fight malaria in a desert somewhere outside of America/Europe; actually, they don't need to count, their exhaustion at the end of a hard day is proof enough of their altruism.

But since the real goals of the EAs are not to do any of that (it's just a bunch of socially-awkward geeks looking to feel accepted in some kind of community and find sexual partners; in other words, trying to feel like normal humans), they fooled themselves into believing that such short term actions are "ineffective"; actually, any actual measurable hard work has to necessarily be ineffective, or else it would actually have to be done. So they keep pushing their agenda to longer and longer term issues, more abstract and less well-defined as possible, which cannot be actually measured, so they can continue to build this weird, meaningless, ridiculous fantasy that they live in (and which supports their social hierarchy) without having to actually do anything concrete.