So what are the current leaders in the field supposed to do to stave off competition? They should convince the public that they do have a doomsday device, claim it must be regulated, and then they can profit from their duopoly because it's exceedingly expensive to break into the high end of the market. The government has its own nefarious incentives, not limited to collecting fees and using the unrestricted versions for surveillance or black hat stuff.
“AGI is just around the corner and could destroy humanity if we don’t solve alignment” is something AI leadership at multiple companies have publicly said.
One camp, using the same mechanism, is trying to make devices that end the world. One camp, using the same mechanism, is trying to save the world.
That nuclear/AI COULD end the world or COULD save humanity is exactly the reason that a truly neutral party would not describe either tech as a doomsday device but would instead describe either as technologies bearing massive potential.
No one building an LLM has ever publically stated that they aim to build a doomsday device and in fact have only ever specifically said that they are trying to avoid doing so. We've no reason to beleive that their interest is any different from any other corporation with agressive shareholders: increasing profits. Dooming humanity leads to lower profits. Lower profits is outside shareholder interest. They will do anything they can to maximize profit, just as any other profit driven org does.
No objective, neutral, observer would apply such loaded terms.
It's okay to have an opinion, but having an opinion makes you...opinionated. Not neutral. The opposite of neutral.
I make no claims as to the need for regulation.
My argument is entirely that the observer described as neutral in the original comment cannot possibly be neutral if their description of the company they discuss is "Doomsday" anything. That's not neutral, it's loaded.
What the comment is describing is an opinionated observer, not a neutral observer. Someone who's opinion is that the company they describe is in the business of creating doomsday devices.
That's a real opinion that real people do have. But, you can't call yourself, or any portrayed character, neutral if they share that opinion.
Our state of mind is a product of our own senses and experiences; we can't hope to comprehend 100% of the relevant facts.
A statement may appear (relative to your experiences & thoughts) to be neutral - but that tells us little about its underlying neutrality.
I do, in fact, think that, if someone says "my technology -- call it what you will, and I won't bother to tell you what it is -- is so dangerous that in the wrong hands or if it isn't carefully contained in a way that I am going to great lengths to try to accomplish it can end the world" -- which is what these people at these companies are saying -- that it would be hard-pressed to not be willing to call that a "doomsday device". I would even go so far as to claim that the exact opposite is more likely true: that to not want to do so requires you to have some extremely awkward bias yourself.
Sure: me using that label to ascribe something related to this product makes some people, seemingly you included, feel bad, and so maybe we can say it involves a "bias", as I clearly can avoid that label, but chose not to, and in so doing colored the discussion in a way that makes you unhappy... but it doesn't make it untrue anymore than any other purely descriptive term is now-a-days often said to be biased. As they like to point out: "reality has a liberal bias" (even if it is just as often liberals annoyed with the terms being used by people ;P).
Again: the bias against the use of that term seems much more fascinating than being upset at the idea that the parable -- which of course is trying to make a point, and so is inherently biased towards that point -- could be attempting to temporary apply it, particularly so as it seems to lead to blindness to what actually happened: people will hear you saying your tech is dangerous, and it will be banned and regulated.
And that's related to why I maintain that I think your example disproves your own attempt at a point: you try to draw a parallel to another technology, as if once we find out it is nuclear reaction research we of course know that that isn't a doomsday device... and yet that is one of the few things that is most obviously to the most people a doomsday device, it is a thing that even the people working on it have always felt might not just be a doomsday device but might be "the" doomsday device (as in, the one that actually does us in), and there has not only been countless people out there worried about what can and will happen in its usage, but it was in fact weaponized and a very very large number of people died due to it, so to try to make the argument that it would be biased to call it a "doomsday device" itself feels very awkward.
But, sure: we can decide that that's biased and throw out the "dunno it sounds like a doomsday device and should be regulated" argument as the argument itself has a bias in it, and that leaves us with... what? A world in which people make these things, talk about how dangerous they are, and then feign in shock and frantically try to backpedal when governments step in to prevent it from being released without careful analysis and regulation and study and planning?
Suppose there is a huge concern—would you prefer they just release it as-is and everyone suffers data-breaches and more supply-chain attacks?
I used Fable 5 for a fair amount of tasks before they pulled it and I can only imagine what an untapped version of that could do.
It’s very capable and equally aggressive in accomplishing its goals.
Yes, because then we get to use SOTA models to defend against the exact same attacks. Fable detected issues in my projects but got downgraded back to Opus before it could tell me about them or fix them. In what world could that possibly be reasonable?
I’m not a security expert by any stretch, and I also want Fable it helped me a lot in a short time, but I don’t like that so many people are throwing shade at Anthropic for speaking the truth. Mythos and Fable are dangerous.
The thing about marketing is that it’s outcome based speech. It’s not really about truth or even feeling, it’s about using the right words to evoke the right outcome. So, we have to think about what Anthropics desired outcome is.
Do they want AI regulation? If so, then they might use language to make it appear as though this new, dangerous technology must be controlled.
Also, as far as priorities and worries go, for most people cybersecurity is way down the list.
Ohhh no it isn't! (Ohh yes it is) etc
"as the general public is far more aware of OpenAI than Anthropic"
I run LLMs on my own gear with llama.cpp (compiled from source) and I could not tell you anything about either company except they fiddle with AI stuff and that (I don't actually care). I glaze over on news about both organisations in equal measure on mention.
I think you'll find that the general public would not be able to name either company without being asked to pronounce their name from it being written down.
I know plenty of people who will put the letters A and I together and get rather confused about what on earth is going on but very few of those would mention any co apart from Microsoft, Google, Facebook or errm Twitter.
No one has a bloody clue about all this stuff apart from us lot and we have no real idea about it either!
Oooh tulips!
HN is a forum and as far as I am aware, anecdata is allowed here. You might disagree with me and my experience and that is fine too but please don't denigrate me.
The recent OpenAI post calls for mandatory safety certification of all frontier level models:
Even in that post he makes some logical fallacies.
Not sure about that one given that the US government just reversed a ban on the exports on the very chips to China that enable said technologies, you don't hear so much about the chip wars any more.
I think entrepreneurs largely approached this administration with the attitude that if you're running Doomsday Incorporated they aren't going to say, "no, don't export that", but "hell yeah baby, how do we get a 25% cut on every sale?" because that's quite literally what they did on the hardware front.
I mean I have no strong opinion on whether Antrophics statement are true or smart, but the idea that it was regulated because someone in this administration thought it posed incalculable risks is a bit funny, i"m pretty sure they wear that as a logo printed on their t-shirts, it seems to be the sole guiding principle of their foreign policies. Palantir has successfully used doom-mongering as an advertisement strategy for a decade or so
I hate to shill them, but wasn't mythos/Fable SOTA?
Your main point still stands without this aside
The dichotomy between Anthropic and OpenAI's treatment honestly couldn't be more obvious. OpenAI has also asked for increased AI regulation, and they've also released GPT 5.5 Cyber which is claimed to have the same vulnerability-finding abilities as Mythos. OpenAI received no such notices like Anthropic. OpenAI also received a government contract, while Anthropic was banned from DoD use.
Regardless of your thoughts about Dario or his company, this treatment is obviously not based in any rational principle, and pretending it is would be stupid.
It's only a matter of months before the open source models achieve this same capability. What is the US government going to do then? Ban all people in the world from accessing the Chinese models? If you think about these arguments for more than five minutes they really do fall flat.
It's indeed quite satisfying to watch them be the first ones burned by the heavy hand of government they worshipped so much.
Instead what happened is a one-off nationalist decree that solves none of the two concerns.
To address the point I think you (and the article) are making; there's a difference between advocating that a scoped, due-process-protected power exist and endorsing any given exercise of it. This point is made in Anthropic's original statement, but it's seemingly been missed by everyone taking a position against them.
>As we have stated publicly, we believe the government should have the ability to block unsafe deployments, as part of a statutory process that is transparent, fair, clear, and grounded in technical facts. This action does not adhere to those principles. [1]
But with respect to your specific post:
>high-minded philosophical nonsense
Taking a sophisticated approach to a problem will generally improve the outcome. I suppose your point is that it can be difficult to deliniate sophisticated & good-faith reasoning from self-serving rationalisation. Much of what I've seen from Anthropic supports the former. Perhaps others could do more to build up the case for the latter.
You can look at things like the circumstances that the company was founded in, the lack of political cozying with the current admin, the lack of controversies or disgruntled ex-employees, them taking a stand against the DoD (against their business interests), positions they've taken on various issues including data centres in gulf states.
>Suppose a company calls themselves The Doomsday Device Company
Anthropic hasn't. What's the explanatory purpose of adding this to the analogy?
>They make and sell excellent-quality doomsday devices | They regularly go online to proclaim that their doomsday devices are the best and most powerful, and also that doomsday devices are dangerous and should be regulated.
Anthropic invests a reasonable amount into assuring the safety of the products they sell. No-one is claiming they sell doomsday devices. Concerns of doom relate to future iterations of the underlying technology.
>The Doosmday Device company spends a great deal of time and money telling everyone: "Our doomsday device is the most doomy of all time!"
They have what's essentially an automated vulnerability-discovering machine. They've done their best to be open about the very real implications of general availability of a system like this. Using the word "doomy" positions them as childish and unsophisticated for doing this, when it's actually the reasonable and responsible thing to do.
Suppose a company calls themselves The Doomsday Device Company. They make and sell excellent-quality doomsday devices. They regularly go online to proclaim that their doomsday devices are the best and most powerful, and also that doomsday devices are dangerous and should be regulated.
The Doomsday Device Company then says they have the world's best doomsday device. (They don't, but they claim they do.)
The US Government hates the Doosmday Device Company for various political reasons, but also has a vested interest in there not being a massive proliferation of doomsday devices.
The Doosmday Device company spends a great deal of time and money telling everyone: "Our doomsday device is the most doomy of all time!" (though it probably isn't) and "Everyone can use it!" (for a lot of money)
It is completely logical, then, for the US Government to say: No, everyone cannot use your doomsday device, because doomsday is bad. (While also meaning: Only we should be able to use it, and you shouldn't be able to tell us how.)
If you do not want to be in the business of having your doomsday devices shut down by the government, well, it would help if you didn't so loudly and aggressively proclaim how doomy they are. It doesn't matter how trustworthy you claim to be, given that your business is making evil doosmday devices. You still won't be trusted!