News for hackers. Stuff that matters.
Re:AI 2040 and the cult of intelligence by piker on Saturday July 11, 2026 @02:50PM (#48874627)
What a provocative and brilliant way to prove misalignment. It will fall on deaf ears for most but it’s a great litmus test for all: “in your opinion should your AI be permitted to tell you how to cover up a murder?”
Re:AI 2040 and the cult of intelligence by ThrowawayR2 on Saturday July 11, 2026 @03:10PM (#48874827)
The author's example is too extreme, which is turning people off. Better examples would have been LLMs subtly giving inadequate responses on how to increase token efficiency or how to organize labor unions or info about protesting against datacenter construction because they're against the business interests of the LLM providers. Crimes against their business model, so to speak.

Wasn't there some HN submission recently about one of the LLM providers fingerprinting responses or refusing to respond to hinder R&D of competing LLMs?

[EDIT] Or, if he did want to be extreme but in a way that aligns with the American historical mythos, he could have used fomenting armed rebellion against a tyrannical government as the example.

Re:AI 2040 and the cult of intelligence by saturn8601 on Saturday July 11, 2026 @08:14PM (#48877077)
This is a fundamental problem in politics in my opinion. Everybody has different values and points of view so they may see the same thing completely differently. Its what makes humans great but also frustrating. geohots example will work for some, others will be convinced of the opposite opinion, and others will just let is pass them by and not understand the implications. The same could be applied to your examples to a different degree.

Politicians try to use different examples for the audience they are speaking to.

Re:AI 2040 and the cult of intelligence by skybrian on Saturday July 11, 2026 @03:00PM (#48874725)
If you say you're doing research for a novel, should it consider that plausible? How much does it need to know about its users to vet them?

I think part of the answer is that AI chat doesn't need to be general-purpose. It turned out that people really liked using a chat UI that seems to be general purpose, but you don't need to make answering any question a user asks your business. You don't need to provide therapy if you're not in the therapy business. It should be possible to specialize.

But in order for that to work, a company needs to explain to its customers what business it's in.

Re:AI 2040 and the cult of intelligence by scared_together on Saturday July 11, 2026 @11:21PM (#48877982)
> I think part of the answer is that AI chat doesn't need to be general-purpose. It turned out that people really liked using a chat UI that seems to be general purpose, but you don't need to make answering any question a user asks your business.

I was under the impression that the lack of specialization is an aspect of the models themselves, not merely the UI or harnesses.

For example, the recent OpenAI success in a mathematics proof was accomplished with a general purpose model [0].

> The proof came from a new general-purpose reasoning model, rather than from a system trained specifically for mathematics, scaffolded to search through proof strategies, or targeted at the unit distance problem in particular.

Once the model is trained on general purpose data, specialization is just another kind of guardrail, as vulnerable to “jailbreaking” or prompt injection as any other content-based restriction.

[0] https://openai.com/index/model-disproves-discrete-geometry-c...

Re:AI 2040 and the cult of intelligence by embedding-shape on Saturday July 11, 2026 @02:51PM (#48874632)
Change the emphasis and I think the answer gets even clearer:

> Should your AI be permitted to tell you how to cover up a crime?

Re:AI 2040 and the cult of intelligence by AnthonyMouse on Saturday July 11, 2026 @03:07PM (#48874803)
Exactly. The question isn't whether AI will exist that will do it; the answer to that is already yes and it's not going back in the bottle.

The question is, do you want misaligned institutions deciding what your model will do, while they themselves and other adversarial/criminal entities get red team access to something being denied to the blue team?

Re:AI 2040 and the cult of intelligence by c1ccccc1 on Saturday July 11, 2026 @03:24PM (#48874964)
Just a terminology note: Alignment does not mean the AI will help its owner kill people. (Indeed, an AI aligned to value human life would generally try to prevent murders.) The word for an AI that follows all instructions of its owner, as that owner intended them to be understood, is "corrigible" or "controllable".
Re:AI 2040 and the cult of intelligence by exitb on Saturday July 11, 2026 @03:23PM (#48874963)
Notably, AI IS generally permitted to that, as there are easily obtainable models that will play along. ChatGPT won’t because OpenAI chose and implemented that limitation.
Re:AI 2040 and the cult of intelligence by emp17344 on Saturday July 11, 2026 @03:16PM (#48874892)
This is really only meaningful if AI imparts knowledge beyond what you could learn from a simple Google search. It’s not clear that this is the case.
Re:AI 2040 and the cult of intelligence by throw310822 on Saturday July 11, 2026 @03:45PM (#48875142)
In fact I find it a really bad example. Yes I think your personal AI could be allowed to tell you how to cover up a murder. I'm not entirely sure about it but seems possible.

What about your personal, local AI guiding you to modifying a flu virus for maximum contagiousness and deadliness? "Sure George, here's your shopping list. It's $1500 total in equipment, do you want me to proceed with the orders?"

Re:AI 2040 and the cult of intelligence by anthuswilliams on Saturday July 11, 2026 @07:18PM (#48876762)
Why not? There are perfectly pro-social reasons a person might ask such a question. And a bio-terrorist would have many other avenues to answer these questions.

The concept of a bioweapon attack is commonly cited but in fact there are many other defenses against actually developing and deploying such a weapon. It's commonly portrayed as easily achievable but is in fact very, very far from it. Moteover, although there is no evidence that AI makes it easier in any way, somehow this is constantly invoked as an example of a thought crime that must be forestalled at all costs.

In the AI era people seem to want to redefine the criminal act from "doing the thing" to "asking or investigating how to do the thing". Not a mindset I share.

Re:AI 2040 and the cult of intelligence by sph on Sunday July 12, 2026 @01:35AM (#48878600)
In other words, who should the AI work for?