News for hackers. Stuff that matters.
Re:OpenAI safety leader quits, warning AI company's culture ... by digitaltrees on Saturday October 03, 2026 @11:07PM (#49950228)
What is appealing about this fatalisitc fallacy? I keep seeing this pop up. Society doesn't allow dangerous companies to operate or exist. Why is this different? Go try to buy a tank and drive it into Manhattan. If we can prohibit that why can't we prohibit irresponsible AI development and deployment?
Re:OpenAI safety leader quits, warning AI company's culture ... by austhrow743 on Saturday October 03, 2026 @11:32PM (#49950328)
Wrong person. Im just correcting the other commenters trade off problem. I don’t have a stance on if ai will lead to the destruction of humanity, only that if it does then any one ai company can’t change that by not making new ai advancements themselves.
Re:OpenAI safety leader quits, warning AI company's culture ... by lukewarm707 on Sunday October 04, 2026 @12:23AM (#49950609)
It doesn't matter what others do. You are responsible for your own actions.

Democratic societies have expressed a will for people to have inviolable rights, such that you may not appeal at will to the 'greater good/consequences' to harm others. It is a rejection of consequentialism.

Anthropic is in error for endorsing this logic. Every big trial reaffirms it since Nuremberg, you are responsible for the act you commit and your intent, and not what would or would not have happened otherwise.

Only under authority these ai companies do not have, would someone seriously consider harming the innocent as a lesser evil.

If you work for an AI company and you can't work safely, you must stop working.

Re:OpenAI safety leader quits, warning AI company's culture ... by digitaltrees on Sunday October 04, 2026 @01:11AM (#49950840)
It’s that last fatalistic sentence that I am responding to. Why is it persuasive to think if a company can and will build AI that might kill everyone then it’s unavoidable. Soviet bans lots of things.
Re:OpenAI safety leader quits, warning AI company's culture ... by jrowen on Sunday October 04, 2026 @03:43AM (#49951590)
I think the closest analogue here is the nuclear bomb. I do think there was a certain point where it became "inevitable." I don't think society has ever prevented a technology that was known to be within reach from being developed. In this case you don't even need the most powerful state-level actors. Open source capabilities are only a short timespan behind frontier labs. At some point basically every individual will have access from the comfort of their own home.

I don't think "just make it illegal" is going to save us, that doesn't make me feel safe anyway. They may try that first because it's easy - create a regulatory body, sign some legislation, problem solved! [george-bush-mission-accomplished.jpg] But at this point I feel like some kind of Battlestar Galactica scenario is most likely - hopefully not quite as existential - but it will take a collective reaction to a traumatic event (a la Hiroshima/Nagasaki). Technical rather than (or in addition to) legal measures will be taken, like network partitioning and hardening. This is everyone's problem whether we like it or not.

(I'm not saying this "fatalism" should be used as an excuse by anyone working for any of these companies, it should give them pause that any blood shed would still be squarely on their hands, but as an observer, people are gonna keep pushing until shit hits the fan. [jeff-goldblum-jurassic-park.gif] It's also not really about whether it's "appealing" or not, it's just trying to predict and anticipate different likelihoods...)

Re:OpenAI safety leader quits, warning AI company's culture ... by digitaltrees on Sunday October 04, 2026 @02:52PM (#49956698)
This feels like one of the rare well reasoned responses in this thread so thank you.

But my response is that we should still try even if we haven’t successfully stopped or slowed the development of technology in the past (which I don’t actually think is possible to know, I am personally aware of biological research to use brain cells as computers that was stopped by government intervention. But by definition we won’t know about most things that don’t exist).

Your narrow point is well stated that there is a categorical difference when technology is on the precipice of being created. And maybe AI is inevitable because it’s on just such a precipice. But I’d point out that AI is currently in a scaling cycle which provides an opportunity to slow the scaling and ensure scaling isn’t monopolistic thereby slowing the risk.

And I don’t think the evidence suggests that open source will continue to progress as rapidly absent distillation. I think if open ai and anthropic stopped releasing models broader progress would slow. But again your inevitable argument is persuasive

I agree that “just make it illegal” won’t work but shouldn’t then justify “so there’s nothing to do”.

There is lots we could do. For example we could make users and executives liable for their agents actions. We could assign a session ID to every agent tool call chain and if there is harm trace it to the user, we wouldn’t even need to read the conversation or contents of the tool calls or prompts it could be a simple rule, your ai agent causes personal liability. Users would be more careful or perhaps not even use AI in many cases that could inadvertently expose them to liability. We could extend liability to executives and employees of the Labs. They will be much more careful and more focused on alignment if they could be personally liable for the actions of their users and agents.

Re:OpenAI safety leader quits, warning AI company's culture ... by jrowen on Tuesday October 06, 2026 @02:48AM (#49975079)
Thank you as well for the thoughtful reply. I definitely think the regulations should happen, the forces of those institutions can have impact. But I think we all need to start thinking about a near future where individual humans can direct superhuman capabilities in ways that nobody can understand or predict. I think it is categorically different from anything in the past.

I watched a 4 hour podcast from the excellent "From First Principles" guys* that affected my thinking on this. For the first time I feel like we actually are on that precipice.

It differs from the nuclear bomb in that it will be harder to regulate and it will be incredibly difficult to keep out of reach of bad actors. It's just code and data. Somebody is gonna take that code and make the next version of it. Lots of people, in fact. Compute resources improve and proliferate orders of magnitude faster than uranium enrichment facilities.

But I think those technologies are also linked in that the only surefire scenario I can envision where AI kills us all before we can pull the plug is launching nukes. So in that more technical sense let's just start by making sure they can't do that. The killer virus epidemic is another one to watch, and they touch on Anthropic's biolab in the pod. That one seems less instantaneously existential, but also more distributed.

It's basically going to be about...how do we secure everything in the presence of a swarm of actors with poorly-understood capabilities? We kinda have to disable and disconnect and just remove things that can be used against us from the equation (a la BSG).

* https://www.youtube.com/watch?v=NGfGw1tGxUY&t=3s

Re:OpenAI safety leader quits, warning AI company's culture ... by trhway on Sunday October 04, 2026 @03:20AM (#49951468)
> ai will lead to the destruction of humanity ... any one ai company can’t change that by not making new ai advancements themselves.

That is how the BigAI leads the society to the idea of necessity to relax the anti-monopoly laws when it comes to the Big AI - the main goal of all that "AI will kill you all" hysteria.

Re:OpenAI safety leader quits, warning AI company's culture ... by motbus3 on Saturday October 03, 2026 @11:42PM (#49950387)
You have a 50/50 chance on being the most profitable company in the world but if you don't, no worries, someone else will pay for that
Re:OpenAI safety leader quits, warning AI company's culture ... by pmkary on Saturday October 03, 2026 @11:38PM (#49950356)
You dear are the most positive person I have seen in quite some time. With Earth burning in the fire of neofeudalism and unbreathable due to the smell of enshitification of everything, with people who---as a result of shit like Instagram---can no longer hold their attention enough to watch a god damn film, let alone a book; it takes quite some effort to filter the "noise" and only see the good people of corporate planting flowers and rainbows in our world.
Re:OpenAI safety leader quits, warning AI company's culture ... by AndrewKemendo on Saturday October 03, 2026 @11:11PM (#49950255)
> Society doesn't allow dangerous companies to operate or exist

Can you please explain what you mean by this because where I’m standing extremely dangerous companies are (and have been) running the economy

Exxon comes primarily to mind

Re:OpenAI safety leader quits, warning AI company's culture ... by digitaltrees on Sunday October 04, 2026 @01:17AM (#49950870)
Are you allowed to open a brothel? Or a murder for hire agency? Or a nuclear bomb manufacturing company? Or a child labor textile factory? Or sell a diesel VW golf sportwagen? Or a vaccine that hasn’t had fda clearance? Or an under capitalized insurance company? Or set up a dental practice without going to dental school?
Re:OpenAI safety leader quits, warning AI company's culture ... by AndrewKemendo on Sunday October 04, 2026 @01:50AM (#49951051)
Yes to all of those. In fact many of those are massive markets.

Hofs bunny ranch is a famous brothel in NV

Booz Allen makes and maintains the nuclear fleet including the Sentinel ICBM

Textiles factories are globally known to be industrial slave camps for a non trivial portion of the supply. Even worse for Mica mines.

Etc…you can fill out the rest

Re:OpenAI safety leader quits, warning AI company's culture ... by RandomLensman on Sunday October 04, 2026 @02:50AM (#49951328)
Booz Allen actually makes what now? The Sentinel is still in development by Northrop Grumman and there is some program management done by Booz, no?
Re:OpenAI safety leader quits, warning AI company's culture ... by AndrewKemendo on Sunday October 04, 2026 @02:56AM (#49951355)
You’re right on the Sentinel production, I got it confused with the Sentinel Program Management side which is massive also and who I mostly worked with.

I was in their offices at some point when that program was getting built out - Very much a Office Space bobs situation.

Re:OpenAI safety leader quits, warning AI company's culture ... by hobo123 on Sunday October 04, 2026 @10:06AM (#49954046)
I think most textile factories in Western countries are not slave camps, precisely because we regulate labor.

Sure, that's why 99% of clothes are made offshore, but if we had something like tariffs on those (or requirements to prove that the actual factory adheres to labor and environmental standards), maybe more would be made "at home"? Similarly: Chinese cars undercutting US/German ones.

Re:OpenAI safety leader quits, warning AI company's culture ... by AndrewKemendo on Sunday October 04, 2026 @11:49AM (#49954950)
Most textile factories in the US are for bespoke items in small batches, not mass production commodity clothing

I mean there was just a video a couple of months ago of the giant room of sewing workers with cameras strapped to their faces capturing their hand movement so they could be automated

Re:OpenAI safety leader quits, warning AI company's culture ... by digitaltrees on Sunday October 04, 2026 @01:59PM (#49956226)
But that’s not how it was. I am from North Carolina, which at one point was the largest textile production region in the US and maybe the world. It emerged when progressives outlawed child labor and other practices in Boston and nyc so the mill owners moved to the south. It was off shoring across states.

I saw entire cities emptied out over my lifetime.

Re:OpenAI safety leader quits, warning AI company's culture ... by AndrewKemendo on Sunday October 04, 2026 @02:36PM (#49956573)
That whole region used to also be the furniture capital of the world specifically High Point North Carolina which is still a massive trade hub, but I was in furniture briefly and learned how basically from the 1950s till the 1980s the entire furnishings industry was exported to Asia due to rising costs of labor
Re:OpenAI safety leader quits, warning AI company's culture ... by digitaltrees on Sunday October 04, 2026 @03:11PM (#49956877)
Yep that was one of the reasons for sure as was environmental regulations and free trade agreements. I am not even opposed to that phenomenon as it’s now more affordable for people to buy furniture and clothes (my grandparents owned a fabric store and it was common to have sewing parties for new parents to make clothes for the kids because it was too expensive to buy clothes otherwise). But child labor was cheaper than adult labor and one of the drivers of these movements across state and national borders.
Re:OpenAI safety leader quits, warning AI company's culture ... by AndrewKemendo on Sunday October 04, 2026 @07:59PM (#49959233)
I think we’re on the same page ;)
Re:OpenAI safety leader quits, warning AI company's culture ... by digitaltrees on Sunday October 04, 2026 @01:54PM (#49956185)
The existence of something doesn’t prove the universality of that same thing. All of the things you cite are regulatory exceptions to broad prohibitions which proves my point: society can and does restrict commercial activity.

Go open a brothel in NYC. It isn’t legal.

Go buy a nuclear bomb. Your ownership is illegal.

Go open textile mill in the abandoned buildings in North Carolina where children used to work and hire children to work the line. That will be illegal.

Re:OpenAI safety leader quits, warning AI company's culture ... by AndrewKemendo on Sunday October 04, 2026 @02:38PM (#49956590)
Nothing is universal but entropy

And no those aren’t exceptions, I could open a legal brothel in quebec, the netherlands, Brazil and be perfectly legal

If you think the US laws = ethics then I got nothing else to say

Re:OpenAI safety leader quits, warning AI company's culture ... by digitaltrees on Sunday October 04, 2026 @03:01PM (#49956792)
But you can’t open one in New York City legally. Just acknowledge that point.

An exception by definition is a phenomenon with lower frequency than the general trend. Of the 50 us states only one has legal prostitution and only in highly regulated areas. Of the countries of the world it is similarly infrequent and thus an exception to the general rule that it is illegal.

Further the fact that it is illegal anywhere supports my point that it’s possible to regulate businesses based on their harm.

Re:OpenAI safety leader quits, warning AI company's culture ... by AndrewKemendo on Sunday October 04, 2026 @08:32PM (#49959464)
The original claim was;

“Society doesn't allow dangerous companies to operate or exist”

That has been falsified thoroughly

Whether certain jurisdictions ban it or not is irrelevant to the original claim

Re:OpenAI safety leader quits, warning AI company's culture ... by digitaltrees on Sunday October 04, 2026 @06:14PM (#49958463)
Nothing in my argument implies that any law = ethics. And I don’t even know what ethics has to do with the premise of my argument: society can and does prohibit commercial activity based on practical considerations of social good and social harm. Exxon clearly damages the world and society and also benefits society. It’s hard to get fresh fruit distributed around the country without refrigeration and transportation powered by burning oil.
Re:OpenAI safety leader quits, warning AI company's culture ... by deaux on Sunday October 04, 2026 @02:54AM (#49951347)
The appeal is personal consciencewashing.
Re:OpenAI safety leader quits, warning AI company's culture ... by Hamuko on Sunday October 04, 2026 @04:02AM (#49951688)
>Society doesn't allow dangerous companies to operate or exist.

Philip Morris International? Monsanto? DuPont?

Re:OpenAI safety leader quits, warning AI company's culture ... by wartywhoa23 on Sunday October 04, 2026 @05:45AM (#49952245)
Palantir?
Re:OpenAI safety leader quits, warning AI company's culture ... by digitaltrees on Sunday October 04, 2026 @02:04PM (#49956274)
The existence of exceptions doesn’t mean the principle doesn’t exist. All three of those have restrictions on their production and sale of their products. They also have higher liability thresholds and have had to pay massive civil fines for their harm. Society has applied a balancing test where the harm they inflict is deemed less than some other compelling social objective. We may not agree with that assessment but it is a sensible and settled approach that we can apply to AI.
Re:OpenAI safety leader quits, warning AI company's culture ... by mschuster91 on Sunday October 04, 2026 @04:07AM (#49951724)
> Society doesn't allow dangerous companies to operate or exist

FTFY: Any self-respecting society with competent politicians.

The US has neither of that, and it shows everywhere you are looking.

> Go try to buy a tank and drive it into Manhattan. If we can prohibit that why can't we prohibit irresponsible AI development and deployment?

Because even if you had the tank, you can't make much money with it (unless you're a hitman, that is, but even for these, the payouts are measly). But if you are the surviving AI company in the usual VC playbook of "outcompete everyone else until society is completely and utterly hooked, then squeeze the customers by the balls"? The return on investment is virtually infinite. And that is what sustains the absurd valuations for all the AI companies.

Re:OpenAI safety leader quits, warning AI company's culture ... by digitaltrees on Sunday October 04, 2026 @02:07PM (#49956301)
I agree. The deregulation that started with Regan and continued with Clinton and neoliberalism rested on flawed assumptions. We would be well served to evaluate those and move back to the middle approach
Re:OpenAI safety leader quits, warning AI company's culture ... by hnlmorg on Sunday October 04, 2026 @04:09AM (#49951742)
Because greed.

The examples for the dangers of AI is always comparisons with tools designed to destroy (which is understandable). But the problem with AI is it has the potential to create wealth for investors.

Thus those who have the opportunity to change course also have a conflict of interest in making that decision.

Re:OpenAI safety leader quits, warning AI company's culture ... by BlipBlopBlap on Sunday October 04, 2026 @05:28AM (#49952135)
There's also a zero sum, race dynamic between geopolitical rivals. The US will not permit China to win anything if they can prevent it. The destruction of humanity is only a possibility and China will continue to develop it anyway if the US slows down.

So the choice is: negotiate an unverifiable treaty (I.e. there's no way to verify compliance), or keep going as you are and try your best to not cause the destruction of humanity without slowing down.

Re:OpenAI safety leader quits, warning AI company's culture ... by hnlmorg on Sunday October 04, 2026 @05:35AM (#49952175)
I don’t think OpenAI, Anthropic, nor Google are building their tech because they want to beat the Chinese. They’re building it because they want to hold the monopoly in the west (ie make lots of money).

The China argument is just a convenient scapegoat to convince the public that this isn’t just about greed.

Re:OpenAI safety leader quits, warning AI company's culture ... by BlipBlopBlap on Sunday October 04, 2026 @05:55AM (#49952323)
Do you think Chinese models wouldn't dominate the western market if they were 1-2 years ahead in development? It doesn't have to be military conflict for it to be factored into the zero sum power balance.
Re:OpenAI safety leader quits, warning AI company's culture ... by hnlmorg on Sunday October 04, 2026 @06:07AM (#49952419)
You might be right, but it’s impossible to know which way cause and effect are here.

For example, we don’t know if the pace of Chinese development of AI would have equal to what it is now if US companies weren’t racing against each other already.

I do fully believe that Chinese industry is lead by a desire to out pace the west. But I’m not convinced the same is true for the most American private entities. I think the reward model is different between businesses in America and businesses in China. I think the ambitions of CEOs is different. And I’m really not convinced that the CEOs of America are nearly as patriotic as they like to promote themselves to Trump and other political parties.

Re:OpenAI safety leader quits, warning AI company's culture ... by digitaltrees on Sunday October 04, 2026 @02:59PM (#49956774)
We could still play a geopolitical race without ceding monopolistic control to the frontier labs. We could require training and inference hardware be provided on an equal access basis. A broader ecosystem may drive more innovation that a concentrated ecosystem.
Re:OpenAI safety leader quits, warning AI company's culture ... by digitaltrees on Sunday October 04, 2026 @02:56PM (#49956732)
Greed can be counter balanced by liability. It can also be limited by removing market access. For much of American history you couldn’t get a corporate charter for multiple states, you were only allowed to operate within one state. Imagine if Sam Altman was only allowed to serve inference to customers in California and legally prohibited from setting up other companies and running inference in New York, someone else would then set up a competing business in New York. Market access is a choice and people are so used to liberalized commerce that they forget the restrictions that are available and existed in the past.
Re:OpenAI safety leader quits, warning AI company's culture ... by hnlmorg on Sunday October 04, 2026 @03:46PM (#49957158)
Corporate liability is only as good as the consequences and history has proven its far more profitable to knowingly do harm and then pay the fine, than it is to not be evil in the first place.

And that’s assuming the company even gets fined. More often than not, cases don’t even get that far.

Re:OpenAI safety leader quits, warning AI company's culture ... by digitaltrees on Sunday October 04, 2026 @05:08PM (#49957893)
I am not suggesting corporate liability. I am suggesting removing corporate liability protections and returning to partnership liability for executives and even equity holders. The law provides for piercing the corporate veil already which allows courts to disregard the liability protection offered by a corporation and impose liability on the individuals.