News for hackers. Stuff that matters.
Re:OpenAI safety leader quits, warning AI company's culture ... by tripleee on Saturday October 03, 2026 @10:23PM (#49950006)
We haven't even built an AI capable of RSI. I don't think the major claim is that it will come via LLMs? Besides- the human brain runs on a tiny amount of energy. Who's to say something smarter than us won't consume just slightly more?

AI safety has been a thing long before LLMs became the focus. Rob Miles on youtube has some really interesting non-doomer non-hypey videos on it all.

> doomers try to predict past the singularity. Isn't that _by definition_ unpredictable

Well you don't need to predict the exact steps that will take place - but you can predict that the AI will want certain things (money, resources, power) to achieve whatever its goal is. Lack of alignment will have it trying to do things we don't want it to.

I can't predict exactly how Magnus Carlson will beat you in chess, but I know he'll do it. Same as if a superintelligent AI exists and has a reason to accumulate things we don't want it to - it's really dangerous to think it won't be able to do it

This topic has been tainted so badly by the AI companies using it for marketing.

Re:OpenAI safety leader quits, warning AI company's culture ... by AlexErrant on Sunday October 04, 2026 @01:45AM (#49951024)
Nick Soares give this chess argument and I was unconvinced. Intelligence is not enough, you also need the ability to manipulate the real world. AGI stuck in silicon won't kill us. You argue AGI will bribe us/divide us/hack us. All possible. I argue that AGI will be set on solving the alignment problem. Also possible. It'll be a race between which AGI wins. It's one nerd's fantasy vs another nerd's fantasy. Soares doesn't _KNOW_ that AGI will "beat us in chess" because AGI changes the rules of the game. Anyone saying they know the probability of solving alignment post-RSI is a liar.

He thinks it's playing chess. When AGI lands, all bets are off: the game fundamentally changes. You can't predict past the singularity. Trying to engage with this fantasy is like a child saying my father can beat up your father. Farts in the wind. My AI can solve alignment faster than your AI can bioweapon us. My made up senario is better than your made up senario. It's fucking stupid.

Re:OpenAI safety leader quits, warning AI company's culture ... by katatue on Sunday October 04, 2026 @03:43PM (#49957129)
Why do you treat unpredictability as neutral? If something has a potentially existential, but completely unpredictable outcome, surely one should avoid heading for it?

Like, knowing that there is a limited number of plausible doom scenarios and what these are, would make me more confident in taking the risk than settling on "it's literally the singularity, we can't know what happens beyond".

Re:OpenAI safety leader quits, warning AI company's culture ... by AlexErrant on Tuesday October 06, 2026 @01:03AM (#49974399)
Perhaps I'm not being clear.

Doom-post-RSI is still very much possible and should be avoided. It is not neutral. If I had a high-variance coin that would either kill or uplift humanity, I would not flip it. The status quo is preferable to extinction.

I am being _descriptive_ in my comments. I am not being _prescriptive_. Nowhere am I telling anyone what to do or what to think: quite the opposite. I am simply saying things that, to me, are obviously true.

I'm perhaps being... loose with my language when it comes to knowing things post-singularity. E.g. we can still discuss scenarios in which the takeoff is fast/slow, or what would be necessary to bring robotics/electricity/logistics to the wider world, what the limits of intelligence are (you can't RSI your way to a bioweapon if you lack a wetlab). That kinda thing. And it's even likely that RSI will be spikey: why would RSI in, say, machine learning, necessarily lead to superintelligence in, say, writing fiction? Or cooking a Michelin-star meal?

My main point is that doomers saying SUPER INTELLIGENCE IS ENOUGH TO KILL US ALL is baseless. That's all.

Re:OpenAI safety leader quits, warning AI company's culture ... by chrisjj on Sunday October 04, 2026 @04:30AM (#49951836)
> We haven't even built an AI capable of RSI.

... that we know of.

Right now it would make sense for anyone who has done so to not tell.