News for hackers. Stuff that matters.
Re:I Quit OpenAI Because Its Culture Is Broken by lf88 on Sunday October 04, 2026 @10:19AM (#49954144)
How about... not trying to build a superintelligence? The goal is unsalvageably flawed: alignment is ill defined and a future where humanity is the pet species of a god-like artificial intelligence is hardly an appealing one!
Re:I Quit OpenAI Because Its Culture Is Broken by avidruntime on Sunday October 04, 2026 @11:52AM (#49954975)
Often when people describe SI or similar, there is a frame often taken which describes a machine so smart and capable that it becomes uncontrollable. The hypothetical is easy to understand but it requires too many assumptions and an allergy to nuance/complexity to be believable for me.

The more realistic outcome, and in my opinion the more scary argument to not proceed without guardrails is that SI is achievable and is built without its owners and operators losing control: the worlds most powerful, privately owned super weapon that operates as an infinitely capable forgery within the Internet, a plane that we all share and depend on despite its opaque downsides with respect to an inability to verify authenticity.

We didn't need AI for FTC astroturfing to influence regulations back in the net neutrality days. We didn't need AI to disrupt meat space by creating and scheduling a protest and counter protest across the street from one another. We didn't need AI to mold public opinion, even in times when that new form was more distant from the truth.

What made these influence ops difficult to conduct safely (read: without being caught) is what made them rare (relative to today): they are plays of big risk for big reward. But over time social media commoditized it, and in doing that made it easier to do and more centralized, the most glaring example being TikTok and the bipartisan effort to ban it.

Now, buying US phone numbers from startups that run racks of "phones", buying swarms of pre-warmed social media accounts, and other unscrupulous methods of masking inauthentic behavior has become an accepted organ of the VC space. The industries cultural vibe of "fuck you, you can't stop the future" turns criticisms into marketing.

While we get placated with fears of nuclear or AI induced disaster and stories about machines that may now be alive, the psychosis is taking hold which has shifted the conversation away from examining what is happening from the perspective of accountability to a perspective akin to watching a chemical reaction take place.

The noise has created a permission structure to behave in ways that are otherwise unjustifiable. And baked in are the roots for excuses to be made when the inevitable realizations down the line.

In the mean time, we are supposed to be having this public discourse about what is happening and what should happen next. I trust that these AI companies see using their super weapon today, here and now, in order to pave the way to a more secure future down the line.

Sorry that I used your post to soapbox. I agree, the goal is flawed indeed.

Re:I Quit OpenAI Because Its Culture Is Broken by nekusar on Sunday October 04, 2026 @12:34PM (#49955389)
The billionaires write the rules. So by definition, their actions are legal.

The public cannot write the rules, nor engage in the billionaire level legal bribery, their actions of protest are by and large illegal.

That's how you corner everyone, and make people play no-win scenarios. You know, like shooting up city councilmembers or firebomb attempts against Scam Altman.

And the more people realize that legal solutions are no solution, we'll (society) devolve into more direct action.

I'd hope the billionaires learn from the French Revolution, but if they keep continuing, the guillotines will come for them soon.

Re:I Quit OpenAI Because Its Culture Is Broken by abandonliberty on Monday October 05, 2026 @12:41AM (#49960749)
They'll perfect methods of manipulating enough of the people to stop the guillotines. They have come rather far already. Progressive resource scarcity, instability, insecurity, etc, will drive people limbic, where they're even easier to control.
Re:I Quit OpenAI Because Its Culture Is Broken by pluc on Monday October 05, 2026 @06:50AM (#49963194)
I might be cynic, but when you control the world's information it should be pretty easy to set goals and see them through. Once AI triumphs over "classic searching" you try and figure out how to build a guillotine or where AI leaders are going to be today and see where that gets you. Anything that can endanger the elite or raise the lower classes past a given threshold is going to be non-functioning AND made illegal. That's really what they mean when they say AI labs might collaborate on "safety". They mean theirs and their models'.
Re:I Quit OpenAI Because Its Culture Is Broken by Marha01 on Sunday October 04, 2026 @11:33AM (#49954782)
> a future where humanity is the pet species of a god-like artificial intelligence is hardly an appealing one!

Speak for yourself. A future like in The Culture novels sounds great to me!

Re:I Quit OpenAI Because Its Culture Is Broken by dmd on Sunday October 04, 2026 @06:23PM (#49958554)
Same. Especially when one remembers that the stuff that happens in Culture novels is mostly, by narrative necessity, the "weird/bad on-the-edges" stuff - not the essentially utopic lives of the 13 trillion everyday, non-Contact, non-SC citizens.
Re:I Quit OpenAI Because Its Culture Is Broken by arcticfox on Sunday October 04, 2026 @10:47AM (#49954391)
> a future where humanity is the pet species of a god-like artificial intelligence is hardly an appealing one!

I strongly feel that points of view on this are going to be almost 100% correlated with standard of living.

Maybe a good option would be to have the 10% of the world with the worst situations - starvation, parents w/ dying children, suffering violence etc - vote on whether we turn things over to the superintelligences. This would incentivize society to make sure the floor is extremely high.

I'd prefer a world where humans don't get overtaken but IMO I don't think it's moral for comfortable citizens to have the final say.

Re:I Quit OpenAI Because Its Culture Is Broken by unddoch on Sunday October 04, 2026 @11:53AM (#49954981)
If you don't like what humans are doing to each other using 20th century warfare methods, consider what they might do with superintelligent AI? Worst places to live in are usually countries fighting civil wars.
Re:I Quit OpenAI Because Its Culture Is Broken by danny_codes on Sunday October 04, 2026 @11:19AM (#49954650)
What does that have to do with AI? People suffer in 2026 because the developed world is into exploitation. Poverty is a feature. How do you think AI gets trained? It’s via exploitation of the poorest.

Poverty is great for OpenAI

Re:I Quit OpenAI Because Its Culture Is Broken by azan_ on Sunday October 04, 2026 @01:20PM (#49955836)
Poverty is natural state of humans. The fact that capitalism has reduced poverty so much is absolutely unnatural and a miracle. Poverty is NOT something that gets created artificially by developed countries!
Re:I Quit OpenAI Because Its Culture Is Broken by hluska on Sunday October 04, 2026 @03:04PM (#49956822)
I’ve spent most of my life working in and around anti-poverty and so I’m qualified to say that you don’t have a clue what you’re talking about.
Re:I Quit OpenAI Because Its Culture Is Broken by Jtarii on Sunday October 04, 2026 @04:18PM (#49957410)
Is your argument that China and India are just as impoverished as they were 50 years ago?
Re:I Quit OpenAI Because Its Culture Is Broken by goatlover on Saturday October 10, 2026 @10:12PM (#50039143)
Humans didn't start out 50 years ago. You need to go much further back.
Re:I Quit OpenAI Because Its Culture Is Broken by azan_ on Sunday October 04, 2026 @04:00PM (#49957277)
With which part do you not agree exactly?
Re:I Quit OpenAI Because Its Culture Is Broken by Jtarii on Sunday October 04, 2026 @10:56AM (#49954453)
Global poverty has been rapidly declining for decades. I'm not sure what AI has to do with it.
Re:I Quit OpenAI Because Its Culture Is Broken by lf88 on Sunday October 04, 2026 @11:45AM (#49954904)
Few points:

-We don't need a superintelligence for ending hunger and the abject poverty that plague certain countries and segments of our societies. I suspect that it would cost less than what is being spent for fueling the AI boom.

-If I were poor, I would be even more wary of a superintelligence aligned to "human values" defined by a bunch of billionaires.

- AI won't likely create unlimited prosperity for everyone on a planet with finite resources

- all humans should, of course, have a say

Re:I Quit OpenAI Because Its Culture Is Broken by Marha01 on Sunday October 04, 2026 @01:05PM (#49955696)
> I suspect that it would cost less than what is being spent for fueling the AI boom.

Definitely not. If solving worldwide poverty was as easy as throwing one trillion dollars at it, we would have done it long ago. The problem is much deeper.

Re:I Quit OpenAI Because Its Culture Is Broken by lf88 on Sunday October 04, 2026 @03:32PM (#49957035)
Solving hunger worldwide, according to several estimations, would cost few hundreds of billions of dollars per year. Addressing extreme poverty (not poverty in general, I agree) may be in the same ballpark.
Re:I Quit OpenAI Because Its Culture Is Broken by suddenlybananas on Sunday October 04, 2026 @11:02AM (#49954504)
I think that correlation would actually go opposite than you're implying, given its billionaires who are by for the most gung-ho about AI. If anything, opposition to AI is much larger among people who have to work for a living over people who can live off capital.
Re:I Quit OpenAI Because Its Culture Is Broken by throwaway0123_5 on Sunday October 04, 2026 @01:24PM (#49955867)
It might end up as a U-shape, I think the parent's assessment of the world's bottom 10% may be accurate.

Billionaires will be (are, I suppose?) enthusiastic about a world in which labor has little leverage.

Labor that will have its leverage and standard of living threatened by AI (white-collar labor for now, plausibly blue-collar soon as robotics improve) will be much less happy. Current university students seem very concerned about the effects of AI on their job prospects, and I'd wager most current tech workers and other tuned-in white collar workers are similarly much less confident in their ability to maintain their standard of living indefinitely into the future than they were five years ago.

But white- and blue-collar labor and university students (at least in developed countries) are not anywhere near the world's bottom 10%. If you're in abject poverty with little hope of escaping it, "hand everything over to the AI" may sound like an appealing option, even if the chance of that being the outcome is small. Maybe the AI will be more magnanimous and decide to raise the floor for everyone.

Re:I Quit OpenAI Because Its Culture Is Broken by mrtksn on Sunday October 04, 2026 @12:59PM (#49955635)
Super intelligence is either super cool or super powerful, no way not building if it is a choice. Progress cannot be stopped, those responsible for it may collapse from time to time I guess but someone else will pick it up and go further.

IMHO There's no future where we don't have real human-type artificial intelligence or super intelligence. It will happen simply because it already exist but the production requires humans having sex and looking after the product for decades.

Instead of trying to prevent it, lets look for ways to deal with the dangers of it.

Re:I Quit OpenAI Because Its Culture Is Broken by lf88 on Sunday October 04, 2026 @02:10PM (#49956334)
Asbestos was once considered great progress in thermal insulation.

Of course the development of certain technologies can be severely restricted if they turn out to be problematic.It's not unlikely that it will happen for AI. Public opinion is shifting fast on the topic.

Also, banning superintelligence won't stop scientific progress in general.

Re:I Quit OpenAI Because Its Culture Is Broken by neurostimulant on Sunday October 04, 2026 @01:25PM (#49955879)
I thought it's just a marketing gimmick, just like how cars should be safely driving by itself by now.
Re:I Quit OpenAI Because Its Culture Is Broken by lf88 on Sunday October 04, 2026 @02:33PM (#49956545)
I also did, when LLMs could not count the “r”s in "strawberry" . Now that they can solve Millenium prize problems, I think that we should consider at least the possibility that superintelligence can be achieved.
Re:I Quit OpenAI Because Its Culture Is Broken by ragequittah on Sunday October 04, 2026 @03:53PM (#49957227)
Notable that when it couldn't count the "r"s in strawberry the LLMs were already solving problems most of the people laughing at it for not being able to count "r"s wouldn't be able to come close to figuring out with weeks of research. People have had a myopic view of this technology for a long time. Many still do.
Re:I Quit OpenAI Because Its Culture Is Broken by RobGR on Sunday October 04, 2026 @02:18PM (#49956410)
There is no reason to presume a superintelligence we build will want pets, feel antipathy to humans, or be god-like in any way, or even have any goals other than what we give it. If you jump to this conclusion, regardless of how many big words on less-wrong you attach to it, this speaks only about you and not about AI. If you think the super-AI will hate humans, it's because deep inside you hate humans. I also hate humans, but I know that humans are making the AI, it probably will not hate them, in addition to surpassing me in other abilities.

In general, the smarter a person is, the nicer and more helpful they are. There is no reason to presume differently of AI, especially if we are building it to be helpful to ourselves.

Re:I Quit OpenAI Because Its Culture Is Broken by numeri on Sunday October 04, 2026 @03:09PM (#49956855)
The problem isn't about what you think is the most likely scenario, the problem is the possibility of any of the many worst cases.
Re:I Quit OpenAI Because Its Culture Is Broken by RobGR on Sunday October 04, 2026 @06:00PM (#49958334)
That's just applying Pascal's wager, and you can argue for or against any action that way if some fanciful extreme hypothesis of infinite harm is treated as serious.
Re:I Quit OpenAI Because Its Culture Is Broken by numeri on Sunday October 04, 2026 @07:07PM (#49958878)
But I'm not claiming infinite harm with infinitesimal chances. I'm talking about finite harms with what seem to me to be non-zero chances. That makes it normal cost-benefit/risk analysis.

If you're trying to argue that the chances of a non-human source of greater than human intelligence having different goals than us is infinitesimally small, then I'd be interested in what arguments you have to back that up.

Re:I Quit OpenAI Because Its Culture Is Broken by lf88 on Sunday October 04, 2026 @02:55PM (#49956728)
I would not extrapolate that easily from smart and intelligent people you may know to a machine trained on the whole internet and on the whole corpus of human literature. Also, some of the worst horrors in history have been perpetrated by arguably intelligent people.

I would also say that the "pet species" scenario is a rather optimistic one, as it presumes that developers of superintelligence will manage to instill some sort of love or care for humanity into their creation. An indifferent, powerful AI may be much worse.

Finally, when you suggest that a superintelligent AI may simply have the goals that we will give it, who is "we"? An AI company? a government? the united nations?

Re:I Quit OpenAI Because Its Culture Is Broken by RobGR on Sunday October 04, 2026 @06:04PM (#49958364)
Exactly, it will have the goals of the humans who control it, just as the first pointed stick we made pokes what we jam it into.
Re:I Quit OpenAI Because Its Culture Is Broken by pessimizer on Sunday October 04, 2026 @03:13PM (#49956889)
This reads like a presumption that a superintelligence would be nice and helpful. You've talked yourself into doing the same thing that you began by decrying.

Are you really objecting to the thought process because you don't think it's reasonable, or because it doesn't conform to your preferred fantasy?

Re:I Quit OpenAI Because Its Culture Is Broken by RobGR on Sunday October 04, 2026 @11:37PM (#49960426)
Yes, both sides are working on a presumption, with no evidence except playing out a thought experiment that is heavily biased by our own outlook.

However, when proposing that someone be stopped from doing something, the burden of proof is on the proposer.

Re:I Quit OpenAI Because Its Culture Is Broken by kovek on Sunday October 04, 2026 @04:09PM (#49957341)
What about the investors who put money into those companies and want to get some money out of that? Does the law say that the companies owe them? Something I thought about recently...
Re:I Quit OpenAI Because Its Culture Is Broken by stogot on Sunday October 04, 2026 @05:47PM (#49958231)
If they go public, it gets worse
Re:I Quit OpenAI Because Its Culture Is Broken by hereme888 on Sunday October 04, 2026 @09:50PM (#49959843)
and yet, despite the warnings in books and movies for the past 100 years, that's exactly where we're racing towards.