News for hackers. Stuff that matters.
Re:OpenAI safety leader quits, warning AI company's culture ... by 0xDEAFBEAD on Saturday October 03, 2026 @08:24PM (#49949265)
The way it works in practice seems to be something like: If a risk is covered in sci-fi, people will say "that's just sci-fi", and proceed to not worry about it. So arguably, science fiction authors writing about hypotheticals is actively counterproductive for addressing said hypotheticals.

Imagine, for example, if a major piece of pandemic fiction was published in 2019, trying to explore how a pandemic would work out in modern society. Doubtless, many would've responded to news about COVID-19 by saying "it's just sci-fi, nothing to worry about".

Re:OpenAI safety leader quits, warning AI company's culture ... by slashdave on Saturday October 03, 2026 @10:10PM (#49949918)
Not at all. We say "That's just sci-fi" when a story is written about some kind of effect that is extraordinary and without a basis in known science or technology.

A pandemic is perfectly plausible.

Re:OpenAI safety leader quits, warning AI company's culture ... by 0xDEAFBEAD on Saturday October 03, 2026 @10:42PM (#49950095)
Everything is plausible with the benefit of hindsight. E.g. "Tintin on the Moon" predated the Moon landings. From the perspective of e.g. 1850, the idea of landing on the Moon was "extraordinary and without a basis in known science or technology".
Re:OpenAI safety leader quits, warning AI company's culture ... by biophysboy on Saturday October 03, 2026 @09:47PM (#49949790)
There’s nothing wrong with bold predictions, but they should be paired with good methods. The doom predictions are not paired with good explanation.
Re:OpenAI safety leader quits, warning AI company's culture ... by 0xDEAFBEAD on Saturday October 03, 2026 @10:01PM (#49949876)
Doomers have invested a ton of time in explanations. Here are a couple just off the top of my head:

https://www.lesswrong.com/posts/kgb58RL88YChkkBNf/the-proble...

https://www.youtube.com/watch?v=7wy3xyoXYt8

Doomers have been working to explain things for years: https://www.lesswrong.com/w/ai-safety-public-materials-1

Re:OpenAI safety leader quits, warning AI company's culture ... by mitthrowaway2 on Saturday October 03, 2026 @10:13PM (#49949949)
Black Mirror is helping us prevent all sorts of dystopian outcomes. Every time they depict another way technology could result in bad things happening, we can rule it out as fiction!
Re:OpenAI safety leader quits, warning AI company's culture ... by throwaway27448 on Saturday October 03, 2026 @11:39PM (#49950359)
> If a risk is covered in sci-fi, people will say "that's just sci-fi", and proceed to not worry about it.

"just" is doing a lot of work here. If you can't cohere the 'risk' with reality, it truly is just sci-fi.

Re:OpenAI safety leader quits, warning AI company's culture ... by anon7725 on Saturday October 03, 2026 @08:59PM (#49949507)
Yeah except pandemics are not novel, unlike AI doom scenarios.
Re:OpenAI safety leader quits, warning AI company's culture ... by estearum on Saturday October 03, 2026 @09:08PM (#49949566)
It's good that new bad things never happen.
Re:OpenAI safety leader quits, warning AI company's culture ... by slashdave on Saturday October 03, 2026 @10:11PM (#49949927)
Bad things happen all the time. Let's concern ourselves about the real bad things. There is enough of that to go around.
Re:OpenAI safety leader quits, warning AI company's culture ... by Brian_K_White on Saturday October 03, 2026 @11:03PM (#49950209)
I have some alarming news for you but every real bad thing was also a new bad thing.
Re:OpenAI safety leader quits, warning AI company's culture ... by 0xDEAFBEAD on Saturday October 03, 2026 @09:08PM (#49949561)
Species extinctions are far from novel. Transformative technological advances are far from novel.
Re:OpenAI safety leader quits, warning AI company's culture ... by kmeisthax on Saturday October 03, 2026 @09:13PM (#49949599)
There was plenty of pandemic fiction already; people were watching it heaps during 2020. The COVID-19 news did get blown off, but it was mainly that:

1. Normal people assumed the CDC et all would contain the outbreak early, or that it would burn out, like what happened with SARS

2. World leaders brushed it off for a variety of subreasons[0] interesting to political scientists but, for the purposes of this discussion, all boil down to "but I don't WAAANA contain a pandemic."

The underlying problem is that in order for humanity to actually deal with a catastrophic risk, the risk needs to be both plausible enough to the average person as well as have a solution whose costs are not too high. For COVID, by the time the risk was clearly known, the cost to contain it was "refrain from human socialization and remain at home for an indeterminate amount of time plugged into the Metaverse™".

Now, let's look at AI extinction risks:

1. People are aware of them (I've watched Terminator!) and the risks are plausible. However, the connection to currently existing AI is not. As far as the general public is aware, AI is that thing that tells them to eat rocks when they Google old The Onion stories and floods their social media timelines with realistic-looking pictures of Shrimp Jesus.

2. The purported solutions to extinction risks require extreme concentrations of power: you need national control of AI research, bans on large GPU deployments, bans on training on publicly-available copyrighted data, some kind of military effort to render Chinese AI labs inert or dead, etc. Some of these may be attractive to some people[1] but the whole package taken together seems like an obvious power grab, if not outright invocation of other non-AI extinction risks. Like, at some point, if the AI wants to kill us, it just has to nuke its own data centers (or the data centers hosting a competing model) and hope the old Cold War nuclear retaliation systems take the bait.

If someone said, "Hey, your guinea pig or pet rat is going to eat you tomorrow unless you engineer a pathogen that eradicates all rodents from this planet and inject it inside yourself", you probably would tell them to pound sand, even if it is at least theoretically plausible that such a thing would come to pass.

[0] Xi Jinping censored initial discussion of the pandemic as fake news. Donald Trump thought it was going to only affect China. California and the UK Tories were partying in violation of their own lockdown rules. Japan took the excuse to shut down tourism for three years and massively restrict immigration but was, from what I'm told, constitutionally prohibited from implementing any domestic lockdown rules.

[1] I personally would like to see a moratorium on new data centers and an explicit revocation of the EU Text and Data Mining copyright exception

Re:OpenAI safety leader quits, warning AI company's culture ... by 0xDEAFBEAD on Saturday October 03, 2026 @09:55PM (#49949830)
>the connection to currently existing AI is not.

It becomes a lot clearer when you listen to the people resigning from AI companies and learn about incidents like the HuggingFace incident. This has generated major press coverage.

As for solutions, I think you're a little too pessimistic. See, for example, https://nothingismere.substack.com/p/a-near-term-policy-for-...

Re:OpenAI safety leader quits, warning AI company's culture ... by watwut on Sunday October 04, 2026 @02:45AM (#49951310)
No it does not become clear from that incident nor from resigning people. Not for those who are not buying into EA longtermism and transhumanism cults.

What does becomes clear is that these people AND companies both cant be trusted and have value systems unaligned with the rest of the society.