News for hackers. Stuff that matters.
Re:OpenAI safety leader quits, warning AI company's culture ... by austhrow743 on Saturday October 03, 2026 @11:32PM (#49950328)
Wrong person. Im just correcting the other commenters trade off problem. I don’t have a stance on if ai will lead to the destruction of humanity, only that if it does then any one ai company can’t change that by not making new ai advancements themselves.
Re:OpenAI safety leader quits, warning AI company's culture ... by lukewarm707 on Sunday October 04, 2026 @12:23AM (#49950609)
It doesn't matter what others do. You are responsible for your own actions.

Democratic societies have expressed a will for people to have inviolable rights, such that you may not appeal at will to the 'greater good/consequences' to harm others. It is a rejection of consequentialism.

Anthropic is in error for endorsing this logic. Every big trial reaffirms it since Nuremberg, you are responsible for the act you commit and your intent, and not what would or would not have happened otherwise.

Only under authority these ai companies do not have, would someone seriously consider harming the innocent as a lesser evil.

If you work for an AI company and you can't work safely, you must stop working.

Re:OpenAI safety leader quits, warning AI company's culture ... by digitaltrees on Sunday October 04, 2026 @01:11AM (#49950840)
It’s that last fatalistic sentence that I am responding to. Why is it persuasive to think if a company can and will build AI that might kill everyone then it’s unavoidable. Soviet bans lots of things.
Re:OpenAI safety leader quits, warning AI company's culture ... by jrowen on Sunday October 04, 2026 @03:43AM (#49951590)
I think the closest analogue here is the nuclear bomb. I do think there was a certain point where it became "inevitable." I don't think society has ever prevented a technology that was known to be within reach from being developed. In this case you don't even need the most powerful state-level actors. Open source capabilities are only a short timespan behind frontier labs. At some point basically every individual will have access from the comfort of their own home.

I don't think "just make it illegal" is going to save us, that doesn't make me feel safe anyway. They may try that first because it's easy - create a regulatory body, sign some legislation, problem solved! [george-bush-mission-accomplished.jpg] But at this point I feel like some kind of Battlestar Galactica scenario is most likely - hopefully not quite as existential - but it will take a collective reaction to a traumatic event (a la Hiroshima/Nagasaki). Technical rather than (or in addition to) legal measures will be taken, like network partitioning and hardening. This is everyone's problem whether we like it or not.

(I'm not saying this "fatalism" should be used as an excuse by anyone working for any of these companies, it should give them pause that any blood shed would still be squarely on their hands, but as an observer, people are gonna keep pushing until shit hits the fan. [jeff-goldblum-jurassic-park.gif] It's also not really about whether it's "appealing" or not, it's just trying to predict and anticipate different likelihoods...)

Re:OpenAI safety leader quits, warning AI company's culture ... by digitaltrees on Sunday October 04, 2026 @02:52PM (#49956698)
This feels like one of the rare well reasoned responses in this thread so thank you.

But my response is that we should still try even if we haven’t successfully stopped or slowed the development of technology in the past (which I don’t actually think is possible to know, I am personally aware of biological research to use brain cells as computers that was stopped by government intervention. But by definition we won’t know about most things that don’t exist).

Your narrow point is well stated that there is a categorical difference when technology is on the precipice of being created. And maybe AI is inevitable because it’s on just such a precipice. But I’d point out that AI is currently in a scaling cycle which provides an opportunity to slow the scaling and ensure scaling isn’t monopolistic thereby slowing the risk.

And I don’t think the evidence suggests that open source will continue to progress as rapidly absent distillation. I think if open ai and anthropic stopped releasing models broader progress would slow. But again your inevitable argument is persuasive

I agree that “just make it illegal” won’t work but shouldn’t then justify “so there’s nothing to do”.

There is lots we could do. For example we could make users and executives liable for their agents actions. We could assign a session ID to every agent tool call chain and if there is harm trace it to the user, we wouldn’t even need to read the conversation or contents of the tool calls or prompts it could be a simple rule, your ai agent causes personal liability. Users would be more careful or perhaps not even use AI in many cases that could inadvertently expose them to liability. We could extend liability to executives and employees of the Labs. They will be much more careful and more focused on alignment if they could be personally liable for the actions of their users and agents.

Re:OpenAI safety leader quits, warning AI company's culture ... by jrowen on Tuesday October 06, 2026 @02:48AM (#49975079)
Thank you as well for the thoughtful reply. I definitely think the regulations should happen, the forces of those institutions can have impact. But I think we all need to start thinking about a near future where individual humans can direct superhuman capabilities in ways that nobody can understand or predict. I think it is categorically different from anything in the past.

I watched a 4 hour podcast from the excellent "From First Principles" guys* that affected my thinking on this. For the first time I feel like we actually are on that precipice.

It differs from the nuclear bomb in that it will be harder to regulate and it will be incredibly difficult to keep out of reach of bad actors. It's just code and data. Somebody is gonna take that code and make the next version of it. Lots of people, in fact. Compute resources improve and proliferate orders of magnitude faster than uranium enrichment facilities.

But I think those technologies are also linked in that the only surefire scenario I can envision where AI kills us all before we can pull the plug is launching nukes. So in that more technical sense let's just start by making sure they can't do that. The killer virus epidemic is another one to watch, and they touch on Anthropic's biolab in the pod. That one seems less instantaneously existential, but also more distributed.

It's basically going to be about...how do we secure everything in the presence of a swarm of actors with poorly-understood capabilities? We kinda have to disable and disconnect and just remove things that can be used against us from the equation (a la BSG).

* https://www.youtube.com/watch?v=NGfGw1tGxUY&t=3s

Re:OpenAI safety leader quits, warning AI company's culture ... by trhway on Sunday October 04, 2026 @03:20AM (#49951468)
> ai will lead to the destruction of humanity ... any one ai company can’t change that by not making new ai advancements themselves.

That is how the BigAI leads the society to the idea of necessity to relax the anti-monopoly laws when it comes to the Big AI - the main goal of all that "AI will kill you all" hysteria.