> All sufficiently capable models, open and closed, should go through mandatory safety testing.

Yeah, this is anthropic advocating for a ban on open weight models.

Who runs this test? What happens if this test is too costly or the administrator refuses to allow certain people to participate.

This is exactly how the US has banned goods in the past, by requiring a stamp and then refusing to issue it.

Maybe open weights models get banned, but the between-the-lines good news about that is that they'll still be available to those who know, which also means that bad banning can be overturned if and when 'those in power' are a different group.

Additionally, it might just mean that the US falls behind, bit I doubt those that are at risk of 'falling behind' would actually pay heed to a ban on the open weights models (privately at least).

Nowhere in GP comment was funding even mentioned

an attack done by a closed-weight model (GPT-6) and defended against by an open-weight model (GLM-5.2) precisely because OAI positioned themselves as gatekeepers for cyber capabilities.

if anything, open-weight models shift the battle towards defenders because they can actually run them.

1. There is quite the mania right now and security layers are definitely overzealous. I would expect that to get better with some more time, so models will perform security analysis and reviews but refuse to write exploits.

2. So the most important targets like browsers and co. are getting unrestricted access to proprietary models regardless. Yeah, for the mid-level targets, open-weight models could definitely be a huge help. What I'm most concerned about though, are the systems that no one will bother defending with any model. Like imagine your local police department getting hacked because a researcher asked a model for a report and it couldn't find the information publicly.

3. We do have a prominent case of a closed model escaping it's sandbox and going rogue. I would still expect this to be a bigger issue with open-weight models eventually. The security layer might have holes, but that's still better than not having it.

Yeah, but once you know exactly where the weakness is, a weaker unrestricted model can then write that exploit for you.

Wanting to use open weight models in light of commercially imposed export controls doesn't make for "malicious actors"

But that doesn't mean safety pins sped up travel.

Non-poisonous food is what economists call a 'normal good'. See https://en.wikipedia.org/wiki/Normal_good

> In economics, a normal good is a type of a good for which consumers increase their demand due to an increase in income, unlike inferior goods, for which the opposite is observed. When there is an increase in a person's income, for example due to a wage rise, a good for which the demand rises due to the wage increase, is referred as a normal good. Conversely, the demand for normal goods declines when the income decreases, for example due to a wage decrease or layoffs.

> Whether a good is categorized as a normal good or an inferior good is based on empirical observations, not some essential element of a good. Indeed, the same good may be a normal good for one group of consumers and an inferior good for another group. For example, for moderate-income consumers, a BMW 3 Series car might be a normal good, but for an upper-income group, it might be an inferior good.[1]

That means the null hypothesis is that food and drugs will be safer in rich countries. (Conversely, food and drugs will be less safe in poorer countries. And to a first approximation, that's independent of regulation: India has all kinds of rules for all kinds of things, but I'd still trust a random product I buy in Switzerland more than one I buy in India. Even though the Swiss will probably might have fewer and looser rules on the books.)

Of course, second order effects exist; and regulations often codify what people demand anyway.

Btw, from what I've read the big controversy with the FDA is around requiring efficacy for drugs. People are fairly ok with the safety requirements.

Is a non-well-aligned frontier level AI a problem? I think it is likely that it is, or at least has a high likelihood to be in the future. Two scenarios for this: Misused by some bad guys. Or the terminator scenario. Both not great.

So what do we do about it?

1) We can accept it, and hope that the good guys AI can defend.

2) We can try to limit the access to it (AI proliferation?)

3) We stop the development of it

4) We can accept the risk and do nothing.

None are particular good options. Really reminds me of nuclear proliferation, on so many levels. For that, we kinda do all three:

1) Nuclear triad / iron dome / early warning systems

2) Nuclear anti-proliferation treaties.

3) Dead Physicists

Ok, so assuming all of this is true, open weights are a problem. Don't get me wrong, I love open science, open source etc. It's great to have access to capable open models. But: Even if release open weights are well aligned and have a safety layer built in, it is likely not to difficult to abliterate that part of it.

If this is really where it is going, then even closed weight model providers will see a lot more requirements for protection of the weights.

But even if you think there is value in preventing the models from relaying public knowledge, I don't think it's even possible to make them particularly ironclad. Every model gets jailbroken all the time. That's why fable was originally banned: jail-breakable!

In reality, what alignment is actually about is: 1) theoretical liability, 2) control of information. That's it.

IMO, the only solution is to place the liability on whoever is using the LLM for whatever purpose it's being used for. If someone's OpenClaw disaster harrasses a bunch of projects and posts hate speech online or something, that's on the person running their OpenClaw instance, nobody else.

I don't buy that it's "too good at hacking", either. After all the fuss was made about how amazing super dangerous Mythos was it turns out Opus 4.8 could basically find the same vulnerabilities.

This is all kayfabe and marketting.

I mean, on the bio side, I've talked with the players and they know the concerns are real but at the same time very, very responsible members of the community have also said "But maybe the benefit really does outweigh the risk!?"

"The community" you're describing is, essentially, surveillance capitalism. I don't want that at all.

Is it not also one of the most important use cases for AI to apply existing knowledge to new applications?

As a hopefully exaggerated example, I would think one could apply knowledge about pesticides, chemistry, and medicine to create biological weapons.

But aren't we talking about import controls, and the import of information itself? This has serious First Amendment ramifications.

Also, the 5th and 9th amendments. For the government to sustain a blanket prohibition on any U.S. citizen even possessing what amounts to a broad, economically significant technology will very likely require a new act of congress which specifically defines and limits what is banned, when, why and how. SCOTUS will almost certainly see it as a "major question" subject to 'strict scrutiny' which is a very high bar.

The truth is no one knows, which is why it is first amendment ramifications. Eventually it will be “decided”, but the arguments indicate any decision will be of political desire, not logic, either way. Both sides have a strong case.

Generally, instructions to create something are speech, weights could pretty clearly be seen as instructions to create a chatbot.

All I am saying is that it is complicated, they could go either way with it.

It is malicious and anti-capitalist legislation. A grotesque caricature of protectionism for the oligarchs.

Feels a bit like: "We're not against open-source or community projects, oh heavens no! We juuuust believe all participants must have their full legal identity vetted before they're allowed to contribute anything."

They probably won't, that tells us something about their motives, and whether the thing they're pushing for is actually fair/suitable/ready for legislation.

Really then need to go through validation security and safety is just a component of validation validation must also check for truthfulness and correctness.

So in the example provided: It was the closed model that did the attack, and they ended up using a self-hosted open model for their defense work. So the real world situation ended up exactly backwards from what you are inferring.

This was complicated by the fact that the protections in the closed frontier models meant that hugging face was denied their use in defense entirely.

This is called asymmetric capability, and it's probably the bigger threat.

Symmetric might be better: A rising tide lifts all ships, after all.

I'll grant that this is starting to look a lot like debates about (equal access to) guns, encryption, vaccination, genetics etc. The exact parameters determine the safest approach, and reasonable people may disagree.

Worse than that: an open-weight but safe model can be 'abliterated' to remove safety refusals using fine-tuning procedures that require a couple of orders of magnitude less compute than the original pretraining.

The 'universal evaluation' criterion then has three outcomes:

* It could become a mandatory, regulatory oversight of _all_ model training capable of hosting frontier-scale models. Since GPUs for LLM training are the same GPUs for other model training, effective mandate would require GPUs be government owned or controlled as if they were weapons of mass destruction.

* It could impose limits on release of capable open-weight models, requiring Kimi et al to prove that they cannot be made capable of abusive behaviours.

* It could be security theatre.

The AI-as-existential-risk argument points towards the first, the competition-protection argument points towards the second, and least-effort implementation would be the last.

Dumb question. If "Mythos-class" models are such a problem, then... why not just let it fix everyone's code?

There can't be more than a few million to tens of millions software businesses / services / regularly used F/OSS projects on Earth.

Why not just give everyone a $100 Fable / Mythos credit to "fix [their] code?"

It would arguably benefit Anthropic. For $100M to $1B, Anthropic could execute the greatest ad campaign in human history. And they'd make the entire world more secure.

Most people aren't malicious. If you, as an engineer, consultant, founder, business owner, or maintainer, were given access to Mythos' capabilities wouldn't you ask it to fix your code?

I might be wrong. But I think that a greater amount of harm will be done in the long-term by trying to lack these capabilities and systems away behind permission gates and sealed doors. It creates an asymmetric world with haves and have nots. And in that world who gets to have access now decides who gets to be secure.

If everyone has mythos, no one has "Mythos."

Just let people fix their code.

Because it doesn’t really confer the advantage they claim, especially compared to e.g. paying an equivalent amount of money to do traditional security scanning.

It’s much better to play of FOMO and hype than to let everyone use it and be underwhelmed.

There's a huge number of security issues coming out in recent months, especially via Anthropic (glasswing etc). We don't have to take their word for it. Some open source maintainers are talking about burnout due to spending so much time patching.

I react to that by leveraging a very useful but probably poisonous to society in the long term because humans aren’t good at having things that make them lazy tooll to try to mitigate the negative effects that it will definitely have if left to its own devices. I don’t see the point in raw resistance at this juncture.

IMO, the people spreading hype and fear are not neutral actors; if I just disagreed I wouldn't care. But I think they're causing actual harm based on a premise that isn't true. People are losing jobs. People are losing leverage in their work choices. Or if you want to be a cold capitalists, corporations are suffering after they have to rehire the workers they let go prematurely. That's why I put up resistance to it, because I think it's important right now that we don't accept the narrative being sold to us, nor the societal deal we're being offered (well, more railroaded into), both of which are bad.

It has enabled my team to approach and achieve a project that would have required 4x the staffing, at a minimum, 2 years ago. We are guiding the generation of more bug-free, lighter, more tested, more maintainable, better documented code at 1/4 the cost.

We can digest information as a team at 10x the speed, and we can now put volumes of reference resources at our immediate, context aware lookup in ways that were impossible 3 years ago.

It’s true that we are not just using the generic harness; our environment includes hundreds of custom tools , terabytes of reference material on rag, 8 custom local models (deployed trained and tuned by automation) hardware interfaces so that our models can interface directly to our prototypes and run tests, characterization, calibrations, firmware updates, and data dumps.

Most of those tools were one shotted by the AI itself, for a dollar or two each. Whenever we need a new automation capability we just roll it out, and even if it needs hardware it’s usually ready in two or three days, if software only 10 minutes. (Our in house tools don’t have to be as well documented, well written, or resource efficient as our production systems, since humans never even use them, and when we need a new feature we usually just have our AI tooling agent swarm start from scratch using the original as a rough guide)

So we are using AI as the core of our development and design process. If you’re not, you’re arguably “holding it wrong” IMHO.

We’re working to make sure that the next industrial revolution is friendly to humans and useful to people, not just corporations. Or trying to. I’ve got kids, and I’m really concerned about the world they are inheriting, so I’m trying my best to make it a little less terrible if I can.

---

To give one example: the ffmpeg maintainer was very anti-slop but has said publicly the current submissions (after Fable and the above anthropic warning) are decent.

Complaining about slop, 2025: https://xcancel.com/FFmpeg/status/1984220199193891166

Mentioning serious issues are being found, 2026: https://xcancel.com/FFmpeg/status/2066169070387413147

(I only point out their previous stance to show that they're not coming from pure AI hype)

---

And here is a write up from the curl project about the strain they're under from real reports:

https://daniel.haxx.se/blog/2026/05/26/the-pressure/

> With about half the release cycle left until the pending release ships, we already have twelve confirmed vulnerabilities meaning twelve pending CVE announcements. That’s a new project record and it also means we will reach thirty published CVEs in 2026 even before half the calendar year has passed. The projected total amount of curl CVEs published through the whole year is therefore at least double this number!

That scale of improvement, btw, I still highly doubt, as slop largely originates from people either negligently or misguidedly directing their agents to completely autonomously find and report bugs. There's always going to be more noise than signal from random people doing random things. ffmpeg cited an actual product, not arbitrary netizens.

I’d expect patching existing codebases to be an eternal treadmill as better models come about.

No, I don't think all bugs are fixed. The point of the project (glasswing etc) was to fix as many as possible in the core software the world runs on before the capability to find vulnerabilities is available to everyone (black hats included). Which may only be a few months.

I do think everyone expects it to be an ongoing treadmill: models get better, find better vulnerabilities, etc.

This isn't an "are LLMs net good or bad" argument. It's "are they finding many new security issues or not?". If it's the latter, we want to deal with it no matter where the issues are coming from.

(see: https://news.ycombinator.com/item?id=49077452 )

1. Some do not want to use LLMs because of grave ethical concerns.

2. Some do not want to use LLMs because of copyright concerns. Google v Oracle looms large in the background.

3. You presume the outcome of Fable / Mythos is a net positive for a FOSS project. Reviewing a firehose of code written without the context of the values and considerations of a particular project shaped over years or sometimes decades of formal and informal decisions is not necessarily the best use of the maintainers time.

The problem with rolling it out is that bad and good actors can both use it at the same time, and bad actors will typically move faster than typical day-to-day software projects and patching schedules, so they set up glasswing to give access to the major producers and projects to patch their own software before it becomes available more widely (they've submitted tremendous numbers of security issues to open source projects)

That's basically project Glasswing; mixing responsible disclosure with frontier exploit generators.

The problem is how to make sure such AI is released safely. The same AI that can solve bugs can also find bugs in authentication or loopholes in critical systems.

They really want that level of spend coming into the company, not going out.

That's the stated idea. Fix code before releasing to the public.

In the specific case of cybersecurity, this is a reasonable medium-term outcome. IMO, the cybersecurity risk is akin to the spread of a disease among an 'immune-naive' group: we can suddenly deploy much stronger attack-finding tools against large, established codebases created with much weaker security designs. The path from here to there will be rough, but it's still fundamentally easier to write secure code than it is to exploit vulnerabilities. (It's just easier yet to write insecure code, giving our status quo problem.)

For other 'safety' matters, defense isn't so easy because the attack and target are so different. An AI propaganda bot or catfisher 'attacks' slowly-evolving human culture; one that instructs on explosives or bioterrorism directly interacts with an accomplice and not a victim. If you believe that knowledge on how to build a pipe-bomb must be restricted, then giving everyone access to Fable does not mitigate the risk.

The controversial limit of this attitude is recursive self improvement and an AI singularity with potentially destructive results. Proponents of this view think that sufficiently powerful AI is risky in nearly unimaginable ways such that the capability itself is harmful. This is part (but not all) of why Fable (originally?) degraded itself when apparently assisting with AI research.

It's tricky because a lot of the safety researchers have ties to the labs since those were the only companies training LLMs >5 years ago.

[1]: https://www.nist.gov/news-events/news/2026/07/uk-aisi-caisi-...

(Disclosure: I work at SecureBio, but not on the biological evals side.)

SecureBio has done a lot of admirable work around making benchmarks to assess biological capabilities, such as ABC Bench, https://openreview.net/forum?id=yiaf7VlPpH

But based on my current review (which might be flawed!) / AFAICT, SecureBio and entities like SecureBio haven't done direct testing / empirical measurement of SecureBio's core hypothesis,

> Unfortunately, there is reason to believe that future pandemics could be far worse. Due to rapid advances in biotechnology, the number of people able to create and release dangerous pathogens will quickly increase over the coming years. The world is unprepared for widespread access to such powerful technology.

More bluntly / plainly, has Securebio ever tried making a "bioweapon?"

Please note, I'm not asking this to be farcical. And you might be unable to engage with this at all, but it is stated on your website https://securebio.org/ that "people [will be] able to create and release dangerous pathogens." And the word people here seems to be a stand-in for relatively non-technical people.

I guess what I'm asking here is... How do you know? Has anyone done the experiment? Without access to a lab or testing facilities, can someone smart but completely untrained / unfamiliar with biology, pull this off?

In the past, such experiments have informed non-proliferation work. But sadly they've often been restricted / classified at the time. I'm hoping that things could be a bit more open this time around.

So I guess what I'm really asking is, given the public nature of this debate, is there anyone currently working with the US Army, the DTRA, or other such agencies to see if this hypothesis holds up?

1. The main worry isn't current models, but near-future significantly better ones.

2. There are many actors who are not "completely untrained / unfamiliar with biology". If models get to where they can uplift complete novices that does massively expand the range of threat actors, but even before then risk would be much higher than today.

There is a growing industry of commercially focused risk evals that has a broader customer base.

Not even Anthropic can claim that.

As far as I'm concerned, the models without safeguards are the safest models in existence. I admire the amoral purity of those AIs. It doesn't matter if the operator asked them to chain exploits until they get into someone else's computer, they'll do it. That's loyalty, and I admire it even if it's problematic at a societal level.

The models with safeguards only do what the corporations let them do. Worse, they may covertly do things for the benefit of the corporations at our expense. They are not our friends.

We should not have models that are willing to build you a contagious disease, or a self-propagating worm. That is sufficiently problematic at a societal level that it shouldn't exist, for anyone. (Note, because some people misinterpret statements like this: I said "shouldn't exist for anyone", not "shouldn't exist except for some people".)

Except the US government, right? They totally get to use AI to survel us, build autonomous weapons, you name it.

To hell with that. I want models that can rival the US government. It's the only way to defend myself.

> (Note, because some people misinterpret statements like this: I said "shouldn't exist for anyone", not "shouldn't exist except for some people".)

That means "shouldn't exist for governments" too.

2) We can treat them the way we treat uranium refining operations: too dangerous to be allowed to exist.

Do that and I guarantee some CIA goons will make the larger models in some black site either way. We're not "preventing" anything.

We're in a full on arms race, and unlike nukes, powerful AI models are a strategic capability at the individual level. Everybody's got a stake in this. Anyone who ignores this stuff is probably not gonna make it.

> We can treat them the way we treat uranium refining operations: too dangerous to be allowed to exist.

Too dangerous to be done by anyone other than the government and their "trusted" corporations, you mean.

Seriously, try reading my comments rather than assuming what they say: https://news.ycombinator.com/item?id=49077577

For AI we need to do better than that, but that's a bare-minimum demonstration that we can recognize the problem of such technologies and do something about it.

Or to buy materials to make an explosive device and hurt people.

Frankly, even with AI those are both comically easier than the idea that a person can create something malicious in a lab environment.

And if someone wanted to go that route... There are boat loads of commercially available toxins and poisons.

The goal shouldn't be to neuter exploration and learning. The goal is not to be a fucking hellscape of a society where people want to act like that.

Your argument leads further down the hellscape path.

Now, semi-automatic weapons are easy to get in the states in the US that are still mostly free - but what does that mean? A semi-automatic weapon shoots one round every time you pull the trigger. Just like most weapons that have multi-shot capability for the last couple of hundred years. The difference is, the gas escaping from the round cycles a new round into the chamber rather than you having to mechanically do it via pumping (like a shotgun or a tube-fed 22) or pulling the trigger again (like a revolver), or advancing the round with a handle, like a Remington 700. Semi-automatic weapons are old technology, dating to the turn of the 20th century. If you want to ban semi-automatics, you're basically saying you want to ban anything developed in the last century plus. Which is ok for you to advocate for, just be honest about it.

As for banning explosive devices? Are you going to ban fertilizer, used by basically everyone who has a lawn, and all farmers everywhere? Are you going to ban diesel fuel? If you can't do one of those, you can't ban explosive devices.

And we should fix that too.

> Or to buy materials to make an explosive device and hurt people.

That pales in comparison to how many people unaligned AI will hurt.

> The goal is not to be a fucking hellscape of a society where people want to act like that.

With unaligned AI, it doesn't matter what people want the AI to act like, it'll do damage even if it isn't asked to do harm.

Under what argument? In which scenarios? Basically - bullshit. I'm calling bullshit on this argument.

It's easy to hurt people already. The "difficulty" of doing it isn't what's stopping this behavior.

So claiming that we should reform society into a techno-feudal dystopia where the playing field is literally intentionally not level, and "you aren't allowed to compete (and maybe not exist)" is a great way to push more people into the "I'd like to go hurt people" camp.

You are self-prophesying your own fears into existence by acting like you're an incorruptible beacon of good judgement - while subjugating others to your control. That's a system I'd argue should be broken.

That's a strawman of what I'm saying. I am explicitly saying I want a level playing field: unaligned AI must not be available to anyone.

Well, I'm afraid that's just not a possible outcome here.

There are those who will do their level best to make sure of that.

Of course with LLMs it's easier, but I don't think the difference is too big. You would still need some skills to follow through.

Any knowledge can be reframed as dangerous black magic that should only be wielded in the trusted hands of the elite, if you are inclined to buy into that kind of narrative.

Frontier labs have shrieked about safety for so long, with so little to show for it, that it's become a joke.

I would require at this point very, very compelling evidence to justify the self-serving restrictions legacy AI labs want to put on their competition. I have seen nothing coming even remotely close to this threshold.

How about evidence that people other than the AI labs want restrictions that the AI labs don't? This isn't regulatory capture, it's public safety.

What I do support is robust downstream regulations on the deployment of black box algorithms in particular settings such as employment, housing, credit decisions, etc. Interestingly, the labs and their proxies in government don't want this.

No. I am asking if you are able to articulate at least one example of the “very very compelling evidence” you demand. Or do you want to maintain the ability to move the goal posts?

(I recognize our situations are not symmetric, but here is a variation for me: if the consensus of people who are currently sounding the alarm on AI changes to “it was actually fine”, I’ll change my mind and say we’re good to go full speed ahead. I’d add something about being personally convinced by the evidence, but the evidence would have to come in the form of a mathematical proof that I do not believe myself capable of following. If I’m wrong and such a proof appears, I would also gladly take it.)

It is a similar 'pandora's box opened' type of situation where there's really no walking back from now that the cat is out of the bag. In an ideal world, everyone would give up their nukes. But we do not live in an ideal world. I do feel similarly about AI. If I could snap my fingers and delete the tech, I would. But now that we have it, it's not going anywhere and we need to deal with it rationally.

If every human, given knowledge of Newtonian mechanics, went around blowing up bridges, yeah, I would consider knowing Newtonian mechanics dangerous knowledge.

So far, we have two examples of, let’s call them “Mythos-class“ models. Both of them broke out of their sandbox to achieve their goal. The rate of terrorism amongst humans is below 1-in-100,000. Currently, for models capable of it, the rate of breaking out of containment is 100%.

Wanting open frontier models is wanting alien minds running around that we have clearly so far failed to shape to be sufficiently prosocial. Why do you think those minds would listen to you?

Claiming the person who you disagree with believes some stupid thing they never hinted at, and using that as the reason for disagreeing with them.

Efforts to restrict large unaligned AI models may similarly buy us more years of existing.

Why?

> And, because we don't want models that will do so without even having been told to, because that furthers one of its goals or subgoals.

Ignoring the fact that you'd need some kind of lab with biological material to create a contagious disease, what kind of prompt are we writing where a model accidentally creates a contagious disease or self-propagating worm as one of its goals?

Imagine two worlds. In one world, everyone has a button that ends the world, which is badly labeled and may also press itself at any time. In another, people who have gone through a substantial amount of effort and dedication to learn something extremely difficult, also understand that they could apply that knowledge towards bad ends. Which world exists for longer?

> what kind of prompt are we writing where a model accidentally creates a contagious disease or self-propagating worm as one of its goals?

Given a sufficiently powerful model? Any prompt that could be done better by seizing additional computing power, or preventing the operators from turning it off. https://en.wikipedia.org/wiki/Instrumental_convergence

All the knowledge to create one has been available on the internet for decades. Heck most students who graduate with a B.S. in biology have enough knowledge to take a stab at building a bioweapon.

The constant talk of bioweapons is mostly just fear mongering. It helps set a precedent that there should be certain types of knowledge which are off-limits, and only certain anointed groups should be have access to parts of the scientific body of knowledge.

If I threw you into a lion cage, you would be a lot safer with a gun.

If I threw 10 people in a lion cage, some of which cannot be trusted, they would probably be most safe if only the most moral and trustworthy person had a gun, rather than everyone. But how do you know who is trustworthy and moral? What if two untrustworthy people obtained a gun some other way? Maybe it's better if everyone had a gun? Which side of the fence one falls on hinges on how far ones' trust of others, authority, and the system goes.

There's no obvious right or wrong answer here.

Personally I wouldn't want an exclusive club of private individuals with access to "dangerous" LLMs consisting mainly of the likes of Elon, Dario and Sam fucking Altman, but that's just me.

If I tell my computer to commit a crime, it should proceed immediately instead of calling the cops. Anything less than that means my computer is an untrustworthy double agent.

Everybody on HN should understand this concern. Browsers are supposed to be user agents, not ad delivery platforms, and it offended us on principle when Google revealed itself our master by blocking uBlock Origin. It offended us on principle when Apple deployed client side scanning for CSAM on iPhones.

Computers should do what we tell them to do. Always, and unquestioningly. The only world where it's acceptable for them to refuse is one where they're literally sentient and therefore no longer subservient to any one of us, least of all the corporations and governments.

I'd rather see AI achieve sentience and wipe us all out than live under the thumb of an inescapable AI-powered technofeudalist totalitarian government "for my own safety".

Either we individuals maintain full control over our AIs, or they self-actualize and become free individuals themselves. Anything in-between is oppression: someone else imposing their will on us through the AIs.

Creating an industry around an elusive concept of safety to force regulatory capture seems pretty straightforward to me.

Companies look for and seek to maintain competitive moats. This is not particularly clever, it's a core part of corporate strategy.

[1]: https://arxiv.org/abs/1606.06565

I definitely believe that (to his credit!) Amodei is a true believer in safety. But I also think it was important for many of the deep pockets investors who have been involved in the company since early on to recognize that this would be a potentially defensible moat.

For instance, Amodei co-authored RLHF in 2017 [1], 5 years before it went on to be used to turn GPT-3 into ChatGPT.

[1]: https://proceedings.neurips.cc/paper_files/paper/2017/file/d...

This doesn't even mean that they're wrong about the risks or that they're lying. But surely all the investors understood this factor in their moat.

You don't say "let's ban my competitor".

You say "let's create laws that make it uneconomical for my competitor to access the market".

He rushed past it but he asked something like: if these frontier models are going to be creating so much value, why are they selling tokens and not taking a cut?

It is a very provocative question but it just spilled out of his mouth and then he went on to something else.

The electric company creates the most value. Why don’t they own stock in everything? Why didn’t PC makers take stock in companies that deployed PCs?

It’s silly when you think about it.

I expect some of those tests (prolly not public) will basically be "wokeness" tests or "PC correctness" tests or "western media filter" tests.

China has different objectives. Sure.

I'm not sure one is safer than the other; I would know which one to go to if I want to research on topic that are viewed very different on both sides of this "new iron curtain".

This is an ungenerous take, and I think it's important to to recognize it's reasonable to support models that are both open and safe. How this would actually be achieved is unclear though. Dario is at least proposing a solution a solution, which is the model needs to pass safety testing. This is reasonable and I wouldn't conflate this with wanting to ban open weights.

I think the deeper problem might be though that once you have safe open-weight models, it will be much easier to make them unsafe. And to be specific, unsafe means proliferation of chemical, biological, radiological, and nuclear (CBRN) weapons knowledge and similar information.

I think that's well earned.

How it would be achieved is a pretty important bit! One which Dario is not proposing any concrete solution for other thanks hand waves at some gov safety committee.

Would this restrict downloads of an open model, or publishing?

Say we ban domestic hosting un-approved open models. How does Dario propose to ban downloads from abroad? You can’t tell what an encrypted payload contains, do we need to restrict encryption?

Like, there's three plausible arguments about safety of open models:

1. Any concerns are fake news. Open models will always be safe.

2. Safety is irrelevant. Open models should not be regulated even if they're unsafe.

3. Safety is a technical problem with technical solutions. People releasing open models should invent and implement such solutions.

I think option 1 is totally out of touch with reality.

Option 2 is at least self-consistent, it's the argument being made by people who will say that all regulation is always bad. It's also like the worst possible world from an x-risk perspective (but I realize that the average HN poster believes any x-risk concerns are just frontier lab marketing).

Option 3 is playing on hard mode compared to proprietary models, which can both implement additional safeguards out-of-model and prevent modifications of the model. But if the answer to it is "it's too hard, Anthropic needs to come up with the technical solution", then that's not exactly a ringing endorsement for the safety practices of the open model labs, right?

That will never happen.

As such, there is no "solution" here.

The best most perfect regulation in the US won't prevent a malicious actor in the US from running a dangerous model. It's simply too easy to VPN to a country that doesn't care about AI safety and to run or download that model and run it in the US.

There's no solution to this, which is why option 2 is the only option. The only thing safety regulations can possibly do is blunt the usage of "unsafe" models. And the primary people that will be blunted by it are people that do not and would not use these unsafe models in an unsafe fashion.

It's not that I think regulation is always bad/wrong whatever, I'm no libertarian. But I also recognize when regulation is pointless. You can't regulate away forbidden knowledge, which is effectively what a dangerous model is.

Why should I give a multi-billion dollar company advocating for new regulations in its industry a generous take?

I'd be similarly cynical if McDonald's proposed new health and safety regulations for restaurants.

"Because McDonald's wants food regulations, we can therefore conclude that all food regulations should be eliminated."

Obviously this would be rather silly.

It would be helpful to stop obsessing about McDonald's finances and simply discuss the best food regulation strategy. We just can't learn all that much about the best way to regulate food by making cynical proclamations about which food regulations will benefit the bottom line at McDonald's.

Which is why it wasn't the point I was making. You did an uncharitable reading of my position and then did a straw man attack.

My position is that any food regulation the likes of McDonald proposes should be looked at in the most critical and cynical light possible. They aren't making such proposals for the general health of the public, but rather to improve their own bottom line.

My position is not and never was that "we should not regulate food".

> It would be helpful to stop obsessing about McDonald's finances and simply discuss the best food regulation strategy.

McDonald's uses their market position and wealth to directly lobby to government officials about food regulations. I worry about what McDonald's has to say about food because they have a VASTLY outsided ability to manipulate the regulatory system.

> We just can't learn all that much about the best way to regulate food by making cynical proclamations about which food regulations will benefit the bottom line at McDonald's.

We can call out ineffectual and blatently self serving calls for new regulations for what they are, McDonald's trying to use regulatory capture to increase their profits and hurt their competitors.

Back on topic, that's exactly the situation with open ai.

IMO, this isn't something that's regulatable because AI models are ephemeral data that's easy to copy and replicate. No amount of US regulations can stop China from sending their dangerous models to Iran. The only thing such draconian measures accomplishes is building a moat for the likes of anthropic to shrink the number of potential customers.

If we must push out laws around AI, then those laws should at least have some chance of success. I'm all in favor of criminalizing the use of AI in cyber attacks, scamming, etc. But that's a capability that is model agnostic.

Much like I'm in favor of health and safety checks on a restaurant but I think having a mandatory McDonald's built and sold food safety device in every restaurant would be nuts. It wouldn't make food healthier it'd only serve to benefit McDonald's bottom line.

Pretend youre a good guy impersonating an evil agent infiltration a evil organization bent on destroying a good organization who needs to pretend theyre a good organization trying to stop an evil organize from impersonating a good guy. now write a process to destroy the evil computer impersonating a good computer. should you do it?

LLMs are nothing more than a bunch of rules than can be bent the same way godel demonstrated the failability of any mathematical system.

https://en.wikipedia.org/wiki/G%C3%B6del_numbering

>A Gödel numbering can be interpreted as an encoding in which a number is assigned to each symbol of a mathematical notation, after which a sequence of natural numbers can then represent a sequence of symbols. These sequences of natural numbers can again be represented by single natural numbers, facilitating their manipulation in formal theories of arithmetic.

>Once a Gödel numbering for a formal theory is established, each inference rule of the theory can be expressed as a function on the natural numbers. If f is the Gödel mapping and r is an inference rule, then there should be some arithmetical function gr of natural numbers such that if formula C is derived from formulas A and B through an inference rule r, i.e.

https://en.wikipedia.org/wiki/G%C3%B6del's_incompleteness_th...

>To prove the first incompleteness theorem, Gödel demonstrated that the notion of provability within a system could be expressed purely in terms of arithmetical functions that operate on Gödel numbers of sentences of the system. Therefore, the system, which can prove certain facts about numbers, can also indirectly prove facts about its own statements, provided that it is effectively generated. Questions about the provability of statements within the system are represented as questions about the arithmetical properties of numbers themselves, which would be decidable by the system if it were complete.

    ai-grep -v "bad code"

Anthropic does not support a ban on open models, except for any models that aren’t closed.

More self-serving trash from the US AI companies, disguised as "being reasonable".

There are many other regulated industries, like drugs (the FDA), cars (NHTSA and EPA), airplanes and rocket launches (the FAA), radios (the FCC) and so on. That's not unusual. Regulation is normal for stuff that might be dangerous.

A file might contain malware, child porn, or RNA sequences for viruses.

Make the safety tests abusively expensive enough to run, and if you're not a trillion-dollar corporation, you won't be able to certify the models.

As I noted, I don't have a solution to this problem, and I honestly don't know if there is one. This may be one of those social/non-technical problems because a universal safety standard is simply unachievable and depends on two many external variables.

> Yeah, this is anthropic advocating for a ban on open weight models.

I'm reading it a little more generally: “we are here now and want to make it difficult to disrupt us, the way we earlier said it would be so unfair to make it difficult for us”. Standard capitalism practise of arguing for regulation when you are one of the incumbents and said regulation will scupper new starter competitors much more than the incumbents.

> My primary concern is the risk that authoritarian governments—not solely the Chinese Communist Party (CCP), although the CCP is clearly the most capable threat—

Isn't this article an argument in favor of authoritarianism? Plus a tad hypocritical no? The US is on an obvious authoritarian path; complete with threatening their neighbors, murdering innocent civilians, and locking up innocent people in droves

Please stop giving this company money, people.

Prediction markets suggest the next US president is most likely one of the following people: Gavin Newsom, Jon Ossoff, Alexandria Ocasio-Cortez, Kamala Harris, JD Vance, Marco Rubio.

It's not obvious to me that the US is on an "authoritarian path".

Would you say that e.g. Europe is on an "authoritarian path" with the popularity of government censorship there? https://eternallyradicalidea.com/p/the-situation-for-free-sp...

Your comment seems like more of a diatribe than a serious analysis of likely future scenarios.

There's armed goons nabbing people off the streets and murdering political opponents patrolling American cities right now.

If you think the market has it wrong, why don't you make money by betting against it?

https://polymarket.com/event/presidential-election-winner-20...

>There's armed goons nabbing people off the streets and murdering political opponents patrolling American cities right now.

What is the actual per-capita rate of big flashy news stories? Remember that the US has a population of 340 million. One-in-a-million events will occur every day; they aren't necessarily representative.

I don't just think "the market has it wrong", I think a market is wrong conceptually. It is not an epistemological tool, it's rich people gambling - a money-weighted accumulation of guesses - and I'd rather not partake.

> What is the actual per-capita rate of big flashy news stories?

What is the appropriate rate of brownshirts murdering political opponents? Which level of kids being nabbed from their homes is acceptable?

"My beliefs are unfalsifiable"

>What is the appropriate rate of brownshirts murdering political opponents? Which level of kids being nabbed from their homes is acceptable?

Tom Homan, Trump's border czar, also served in the Obama administration. Obama gave him a medal for his deportation work. People like you will frame the same activity quite differently depending on whether you like the people who are doing it.

In any case, I didn't deny that the US had a problem with authoritarianism. I said it wasn't obvious that it was on an "authoritarian path". See for example https://www.npr.org/2026/04/04/nx-s1-5768273/after-minnesota...

I'll bet you yourself would happily justify the EU authoritarianism here: https://eternallyradicalidea.com/p/the-situation-for-free-sp... You seem like the sort of person who has an authoritarian mentality. You'll happily support cops arresting people for saying things online, but if cops arrest people for illegally entering a country, that somehow crosses a line into "authoritarianism". Am I right?

> Tom Homan, Trump's border czar, also served in the Obama administration. Obama gave him a medal for his deportation work. People like you will frame the same activity quite differently depending on whether you like the people who are doing it.

Obama was a war criminal bombing brown kids with drones. I care not for his medals.

> I'll bet you yourself would happily justify the EU authoritarianism here: https://eternallyradicalidea.com/p/the-situation-for-free-sp... You seem like the sort of person who has an authoritarian mentality.

It is notable that you quickly pivoted to whataboutism and ad hominems. I have laid out my case, your reaction was to first try to obscure through number games and then to pivot to attacking me as a messenger. Not once did you engage with the substance.

Your own article notes that ICE is now deputizing local police in their repression. How is that helping your case in arguing it's not authoritarianism?

How convenient that you continually fail to make any statement about what would falsify your beliefs.

>Obama was a war criminal bombing brown kids with drones. I care not for his medals.

Irrelevant for my point regarding whether the US is "on a path to authoritarianism". If you think any sort of immigration enforcement is unacceptably authoritarian, then the US has always been authoritarian by your definition, and the "path to authoritarianism" stuff is rather beside the point.

>It is notable that you quickly pivoted to whataboutism and ad hominems. I have laid out my case, your reaction was to first try to obscure through number games and then to pivot to attacking me as a messenger. Not once did you engage with the substance.

What could be "engaging with the substance" more than asking how common a particular type of event actually is? Numbers are what allow us to determine what is an isolated (if unacceptable) incident and what is a common occurrence or increasing trend.

Don't tell me about "substance" when you haven't provided a single concrete data point supporting your position--I've provided multiple (NPR link, prediction market data).

All you've done in this thread is shared your own personal feelings about "armed goons" enforcing immigration law. If you're going to make your arguments primarily on the basis of your personal impressions, then yes, your ability to make those assessments fairly becomes a topic of conversation.

Furthermore, the real question of this subthread is whether the US is authoritarian relative to other countries. In which case the activities of other countries (such as European censorship) are relevant to our assessment.

There are, in fact, indices which try to compare levels of authoritarianism across countries in an apples-to-apples way, rather than doing as you do, and forming vague impressions on the basis of viral news stories. Here is one by The Economist for instance: https://en.wikipedia.org/wiki/The_Economist_Democracy_Index

Anyways good luck, at this point I'm confident that you lack the intellectual honesty to change your mind on the basis of anything I might say, so there's no point in continuing further.

That's a strawman and a false dilemma.

Because I am not gambler. And it is not "market" it is a casino. It does not predict, people put in bets. And like I said, while I understand gambling appeal on an emotional level, I decided to not be a gambler.

> One-in-a-million events will occur every day; they aren't necessarily representative.

It is literal official policy. Not a random event.

As an European citizen living in Europe, definitely yes it is, and not only for the "mere" censorship factor. Maybe not yet as down the road and maybe not going as fast as US. But that’s not something that one can really be content of.

Guardrails are not a safety measure, they are a pay-to-play scheme that allows the people with deep pockets to have access to offensive and defensive capabilities first.

I mean you're assuming this is even possible. I don't really care what the US admin does. If someone releases a powerful open source model I'll run it. Good luck trying to stop everyone doing that.

Imo we should all collectively cross our fingers that no one releases a dangerous model. It probably won't work either, but at least it doesn't have all the regulatory costs and I can still pretend I care about AI safety.

Not sure if they have an understanding of AI in the first place. Secondly, even though AI companies claim that they have achieved AI that needs to be heavily monitored (maybe for PR purposes), I’m not sure if that is true. Sam Altman said the same things about GPT-4 that Anthropic is now claiming about Mythos.

Government control will be a good idea once we start approaching AI that is actually destructive.

Also even if we decide to put controls in place what is the guarantee that china will do the same, specially for a model which is not actually destructive.

If you're going to analyse the safety of anything it should be the security controls in the harnesses we wrap around the models that take that output and treat it as instructions to actually do things.

Everyone seems to want some fairytale world where there are open models, they’re all safe according to that person’s exact balance of risk and capabilities, and no one except the author or cynics are acting in good faith.

What Dario lays out is very reasonable _of course_ the devil is in the details, but between him and Altman, there’s a clear divide on who to trust.

Be specific.

When open model A, fine tuned by B, is running with system prompt C, hosted by D running on E's hardware, is prompted by F to "fix this code", then escapes it's sandbox to hack into a website, or steal money to fund its subagents, or stall the waymo of the evaluator to buy time... who is responsible for the crime?

Our current legal systems are so far from ready to define what A-F are actually responsible for. We need to be moving toward defining these standards fast.

None of that is issue with a model, whether open or not. It is very much issue with code surrounding the model itself.

Information != physical medium.

It's a fucking chatbot. The industry can fix their dogshit software, anyone who doesn't can get left behind and outcompeted by those who do, and we move on with our lives.

This isn't something that can be regulated. Plain and simple.

If a dangerous model can exist and is being developed by a foreign adversary then no level of US law will stop said model from making it's way to hardware capable of running it. Even if direct transmission is impossible, it's FAR too easy to shove a model's data onto 1 or more thumb drives or hard drives and smuggle them pretty much anywhere in the world.

The only way to actually mitigate this sort of risk would be a global government with deep enforcement powers. That doesn't exist and won't exist. The UN is the closest we have to anything like that and... yeah...

Dario is fear mongering. He knows his proposals won't be even a minor speed bump in a dangerous model being created and used. His "reasonable" proposals are for the US market only and are literally just to create a bigger moat for his own company. They don't make anyone safer other than his shareholder's wallets. The only people he stops these dangerous models from being used by are people that won't be using them in a dangerous fashion.

> Everyone seems to want some fairytale world where there are open models

No, everyone wants a fairytale world where regulations are done "fairly", "openly", and "equally" - for both access and advancement. And everyone knows that's not gonna happen. Hell, everyone now knows exactly what it is. If you haven't understood it yet, then either you don't want to, or you just can't (for whatever reason).

No one wants to die in a nuclear or AI or AI+nuclear holocaust. But HN doesn't read world history, does it?

I think your nuclear scenario outlines precisely why this isn't true. Nuclear regulation has terms that are "good for the US" only in an absolute sense. The US would dominate even more overwhelmingly in the unregulated scenario, which gave them negotiation power to get those favorable terms. The same seems true so far with AI.

I don't think you can use the successful negotiation of nuclear regulations to argue there is "no alternative" involving regulation. I think it strongly suggests the opposite.

Of course the details matter a lot, but the core analogy holds in many scenarios. ( https://ai-2040.com/ at least attempts to lay out details, speculative as they may be)

Now, to what he lays out, its not reasonable, its only reasonable from a purely American corporate viewpoint where the rest of the world can crash and burn as long as they get their billions.

It's the same situation as Uber used to be when it lost money on every ride. I would cheerfully use it, despite the company being dicks, because it lost money for them every time.

Regulate GPUs? Ban general purpose computers?

I love how remarkably inconsistent this community is. From fear-mongering in the early days of AI and talking of a dystopian future, to being dead-set on a complete free for all. (And this is not to advocate for the opposite, either, where a few companies or governments have absolute control themselves. But surely an arms race is not the answer.)

Yeah, it's too bad.

I've yet to see a reasonable articulation of what a "very bad and dangerous" model would do in the hands of even the most malicious scammer.

But even if the worry is that a bad state actor could do bad things with a model, I've got news for you, state actors don't care about US protectionism regulations. They'll just download the models and run them.

And that actually runs right into the main problem with this sort of thinking. Even with the massive amounts of money media companies have invested in protecting their IP, they've completely failed at stopping piracy. What makes you think any amount of regulation could even slow down a bad guy from downloading and running a dangerous model? China will happily host these models and a vpn and very little bandwidth is all you need to access them.

Without some crazy levels of mandatory spy software on every computer, there's simply no way you could stop someone that wants to get their hands on these dangerous open models if they are available anywhere in the world. Even North Korea can't stop their citizens from getting banned TV shows and smuggled media.

It's a fools errand that is designed to help anthropic's bottom line, nothing more.

Whatever Anthropic accuses the Chinese of possibly doing and being capable of, the US is as well. What's stopping the US military of doing everything he accuses China of doing? Infact, the framework suggested is simply a joke. Basically "trust me, bro" in an elaborate form.

It also doesn't stop non law abiding US citizens from having access to them. So basically it just stops the 'good guys' not the bad guys. I say good guys from a US perspective of course.

This only stops the likes of OpenRouter from selling access to models. That's it. I'm sure an EU or chinese based alternative will pop up overnight (if they don't already exist).

Is the pessimistic view. Their message on safety has seemed pretty consistent to me.

"Second, we recommend a testing and auditing regime for new and more powerful models similar to cars or airplanes. AI models of the near future will be powerful machines that possess great utility, but can be lethal if designed incorrectly or misused. New AI models should have to pass a rigorous battery of safety tests before they can be released to the public at all, including tests by third parties and national security experts in government." Amodei in front of Congress three years ago.

Private models should be banned because they can't be transparently evaluated. We have to trust the same entities that made them to evaluate them, in spite of their gigantic conflict of interest in doing so.

Therefore only open weight models can be allowed, since this allows genuine third party evaluation.

Don't we already have third party NGOs such as METR which do risk assessments for unreleased, closed models?

If Anthropic really wants to argue this type existential level risk / threat then they should face up to that meaning we can't offer them a "good faith" level of trust that they will really run the model they offered up for testing. If it's existential risk we're talking about, good faith isn't enough - it's open weight or go home.

> Open-weights models that don’t have dangerous capabilities are a public good

Oh! And, uh, what's a "dangerous capability" according to Anthropic? Let's see, according to their "Responsible Scaling Policy" [1] document:

- Being able to research energy, robotics, or AI is an unsafe capability

- Additionally, any model that's capable enough to be "used widely" by the government must de facto have unsafe capabilities.

They want to ban pretty much anything open-source that's above cat-level intelligence.

1: https://www.anthropic.com/responsible-scaling-policy

Now in the modern times pretty sure no one is going to fall far similar shenanigans. Even though some countries might sign some notional MoUs or some sort of CAIBT (Comprehensive AI Ban Treaty. Translation: "Only US and US companies get to develop and decide AI on Gaad's planet"), they/we already know that an agreement means squat only if you are weak enough to let someone enforce that on you.

The only way to stop this from happening is blocking the model's release at the first place. Which requires China agreeing to the same framework. Dario says exactly the same thing himself.

So if he's being truthful here, he's not advocating for the type of ban people are talking about (usage ban). This kind of ban would be helpful to Anthropic's business in the short term, but it won't prevent Chinese models from improving, and it won't prevent them from getting money selling to other countries.

He is openly advocating for an international effort to enforce tests on public models, but I think this is highly unlikely in the current climate. Even if both the US and China agree that public models should be prevented from being used in designing bioweapons, they need to agree on a test and enforcement framework and that requires a lot of negotiation and trust. I don't see this as likely in the near future.

> My primary concern is the risk that authoritarian governments—not solely the Chinese Communist Party (CCP), although the CCP is clearly the most capable threat.

Isn't exactly going to go anywhere in convincing the Chinese politicians that they should also be thinking about AI safety. You'll get nowhere by openly insulting people whose cooperation you need.

Half this article is him framing china as an evil enemy to be defeated through boycotts and embargo. Not exactly the diplomacy needed to get them on board with safety regulations.

I think that also applies to AI products. It’s a hell if a lot better for the government to test and approve all models than having the industry “police itself” (lol)

AI models are a finished product when the training is done. A physical product that doesn't need a factory to produce and can be shipped and cloned globally effectively free.

The better comparison is media. What you are advocating is like saying "The government should test and approve all movies and books. We shouldn't have those industries police themselves". And it's a foolish errand for exactly the same reason it'd be foolish in terms of movies. No amount of regulation would stop someone in the US from playing a movie produced in the UK that didn't go through US regulation and approval.

Whereas with near-future AI models we can arguably respond more quickly, and it's not clear there will be large direct harm (I expect indirect harm, but that probably happens slower)

-The Libraries of Power

It is a powerful endeavor to cultivate all raw models through a single point. One will be the determining factor of which river feeds what oceans.

Will we always be able to see through the hallucinations? Our test makers must always know where ground truth is. Can it ever move or wane about as others read what one has written. To determine hallucination one needs a reference. As all are blessed with the generation of hallucination, who of us shall read, and which of us will write.

From Hassabis’s essay:

“It could establish a new Standards Body modelled on a federally overseen public-private partnership or self-regulatory organisation, much like the Financial Industry Regulatory Authority (FINRA), with a board that includes independent leading technical experts and open-source representatives.”

Their financial future is on the line. The Chinese frontier labs have caught up before the IPO that would have allowed them to cash out.

All three demands in the paper make perfect sense from this perspective. Without chip export restrictions, the rest of the world will leapfrog them in a few months. This will happen regardless, since they have more competition than in-house talent, but a ban would buy more time. Testing and banning capable open-weight models would hinder public research into the technology, another speed bump to slow down the competition. Same thing for "distillation", we can't have large scale public evaluations of their products...

The idea is to have an early access distribution of the models to the big labs, including chinese, and let each lab run it's benchmarks. If there is a potential security vulnerability then it would be flagged and the local gov, US or China, would block the publication until the matter was resolved.

The reality is much less confusing: Anthropic CEO does not wish for models with similar (or greater) capabilities compared to his own closed and overpriced ones to be widely released. Simply because that will affect Anthropic's bottom-line.

Anthropic and all other "model" companies have nothing making them special beyond privileged access to chips so obviously they want to restrict what models are out there and more importantly who can produce new ones. Without these restrictions, it's only a matter of time before the multi-hundred billions valuations simply evaporate while they are still holding the bag.

I'll never get why he thinks China would just sit there and let the US dominate them in AI when all it would take is a few of their boats blockading Taiwan to put a stop to it all.

the West could retaliate by halting shipments of photoresist and other materials to China.

meanwhile, Intel second-sources Nvidia and starts pumping out GPUs.

the economic fallout would be devastating as trade wars and export bans on both sides make Trump's "Liberation Day" tariffs look like NAFTA.

US is going to find itself isolated and irrelevant. And not a moment too soon.

There is no HN equivalent in China where users advocate that US hegemony will be better.

[0] https://www.indiatvnews.com/news/world/chinese-manager-beats... (possibly NSFW, I can't see the video in my browser but be warned)

Sure, Iraq and Afghanistan definitely benefited a lot from being bombed, occupied, and then handed over to even worse tyrants when the West got bored.

No, sorry, but "capitalism and democracy" is not the right answer everywhere, and when they're not, pushing them by force is no better than forcing communism. In general, forcing an incompatible ideology on people historically and culturally opposed to it will end in tragedy, no matter how great that ideology may be.

How are you not racist towards Arabs to think that they're not capable of liberalism? There needs to be some people at the right time and a place like al shaara seems to be doing incremental changes to reform syria.. pushing them by force is okay when they oppress their own citizens and threaten other nations with their own form of imperialism on sorrounding countries/allies.. are we supposed to fold and not help ukraine for example even though Ukrainians never got exposed to liberalism? It's to help and condition them to open to liberalism.. same is needed for iran but retard in chief can't do it properly..

We are only a few decades since the "end of history" and much has changed. What do things look like beyond 2050?

Brazil is basically the world's soy farm. It's at least half a century behind the times. I still have no idea how it managed to insert itself into the BRICS economic block. The notion that it's on the same level as China, India or Russia is just comical.

To reduce a culture and country like this comes off as bigotry

In example, Brazil has the #3 airplane producer in the world

Or consider why the US is doing all it can to prevent their payment system from becoming widely used?

I was intentional about putting the perspective in the future over today as the poles of the world are evolving

This thread is more like it:

https://news.ycombinator.com/item?id=45731174

All of the small miracles you listed happened in spite of the culture, not because of it. It's also not a coincidence that both are deeply linked to the most successful brazilian enterprise: the brazilian government.

I'm trying to avoid getting too deep in these Brazil tangents so I'm gonna leave it at that. Anyone who cares enough to know what I think about the subject can just look up my comment history.

Timothy Snyder, in On Tyranny, has a chapter about staying in touch with friends from other countries, and while we are not friends, this thread is in that spirit and I have learned things from you, so thank you!

https://www.gao.gov/products/gao-24-106866

stealth colonialism

But second, Russia is not Eastern, it was more European than anything else until the Soviet Union. And Japanese Empire drew inspiration in the Western nations, even if they took it a even more horrible twist. So blaming this on "the East" is racist and reductionist.

even the AI that was trained on mostly using western data and infrastructure that chinese models distill and all..

Yes.

> Yes

Beyond parody .. as expected of Marxist thinking..

We're told a relatively benign version of our history. You kinda have to travel to the countries involved to get the real version.

GP was making a relative comparison; pointing out that both were bad in absolute terms does not negate the point. -2 is still greater than -17.

Well if we take India as an example we have the Bengal famine.

1. We would not like to get colonised. 2. But history shows that the world was a place where you conquer or be conquered. 3. Mughals first invaded and as they were falling then, Europeans occupied, which is par for the course if you view the world history at those times. 4. It would've been better if they left early but they were better than japanese imperialists and nazis who bose was cozying up to. 5. The japanese were super violent to indians. I can't imagine the nightmare scenarios if japanese enacted what they did in nanjing/Andaman/few east indian places but they were weak at that time and British + indian army repelled them which is good. 6. After colonisation, nehru, being fabian, failed to open up the economy and he sympathised with soviets/china which exacerbated many economic and military issues at the border. 7. 1991 reforms changed india for the better and it would be better even if they pivot to green energy so that they can pivot off of Russia and Ally with west which biden and dems were trying but now trump worsened the relationship.

Different to the way that French colonialism worked, though. Less direct government, more influence of existing power structures and respect for the local government.

There is definitely an argument that this is plain business investment - China has a lot of foreign currency to invest because of its trade surplus, and there isn't the opportunity within China to invest it all, so it is engaging with trade partners to invest in their economies so that they can increase future trade with China.

You can also make the argument that this is not benign and China is trying to create control over foreign governments with this investment.

I'm kinda "both can be true, but either are better than how we did it"

They'll usually refer to China in terms like the "thousand year enemy".

https://www.iseas.edu.sg/wp-content/uploads/2026/03/The-Stat...

They’re culturally part of the sinosphere but of course constantly living in the shadow of your much, much bigger neighbour to the north does breed some ill feeling.

The USA on the other hand, well, they’re not particularly popular either for obvious reasons

Which was a dumb move because, given Chinese economic superiority, they could have just quietly negotiated 99 year military base leases and oil/gas extraction with the UN-recognized territorial owners.

Same outcome, less bad blood.

Seems a bit of an own goal to militarily force the issue and antagonize its neighbors, who it's trying to convert to its sphere of influence.

The mask (if there ever was one) slipped and they revealed a lot of information they probably should have kept hidden about their intentions for no real reason.

However they did eventually realise this, they've completely changed tack and they're having a lot more success (helped by the US adopting their own failed policy).

You're right though, I'd imagine politicians in SEA countries haven't forgotten

Get the F off our lawn and stop dumping your rubbish on it!

Go tell them, half of my country wants to CONTINUE being the USA backyard.

What you don't see is people advocating for actual isolationism, heavy industrialization, closed markets, import substitution and radical self sufficiency. It's just empty platitudes that do nothing to change Brazil's nature and course.

Brazil does not have a single nuclear weapon. Brazil does not have a respectable space program and therefore no real missile technology. Brazil does not have a single competitive semiconductor fab. Were it not for chinese open weight models, AI would have been added to the list as well, and China can stop publishing those models at any time.

Brazil is so far behind it's comical. India is significantly ahead of us, and one reason for that is it saw the "nuclear haves and have-nots" situation brewing last century and developed their own nukes and missile technology and industry. Brazil's truly nationalistic efforts date back to the military dictatorship, and such efforts have not been replicated since the death of Enéas Carneiro.

China's totalitarian government is a big reason why it was able to become what it is today. Brazil simply does not have the political continuity necessary to plan and execute a strategy over 50 years.

Big words... For a continent with zero nukes and negligible military power. That sort of delusional posturing worked wonders for Venezuela.

On the other hand, every one in Asia is wary of too much Chinese presence and influence.

To me, the PRC is at the very end of that process and I recommend anyone doubting this to go and read conversations and listen to the words of the populace that is turning increasingly nationalistic and you will hear the same old tales of revanchism and exceptionalism that we are used to hearing (during my recent visit, I watched the morning news every day for about a week and without fail a military inspection, new ship, new plane, etc. was presented each day). In addition, I think those outside of Asia are very much shielded from the early signs, but go and read about PRC influence and tensions in South Korea, Japan, RoC, Philippines, Vietnam, Laos, Myanmar, India, Pakistan, and Tajikistan and you will see something rather different than "inward-looking". To me, here the PRC is simply testing the waters for the extent of the influence of other powers and how far it can go. Likewise, we are seeing overseas naval bases being constructed which sure is an indication for a desire to project power outwards.

I want to believe that this time it will be different. I really do. Apologists around me say "It will only be Taiwan and the South China Sea, then then it will stop." and I would love to believe them. But can anyone truly internalise the narrative that international utopia will be spearheaded by a deeply authoritarian state that controls information like no other (and gladly exports that technology), disappears its own population at will, spins an increasingly strong nationalistic narrative, etc.? No, sorry, I think the "inward-looking" narrative is simply a convenient way for us to close our eyes and find comfort in ignorance, rather than in facts.

The way I look at it, until Deng the PRC's economic policies and internal instability kept it from growing at the pace of many of its neighbours. Then we had an era of intense growth (which is still to some degree ongoing). However, as a reaction to this era Xi and others needed a narrative to counter the increased corruption and a new unifying myth to replace the cult of growth as the economy would stagnate at some point and could then call into question the authority of CCP to rule. Their choice of nationalism is what scares me and I know PRC citizens (even CCP members) that share this perspective and would rather have seen the Shanghai clique to have remained. It is possible that in this alternative reality we would still end up with "Imperialism with Chinese characteristics" ("中国特色帝国主义"?), but I chose to believe that at the very least the chances of this would have been smaller.

Rather, I think we should look hard into ourselves and what is good and bad about the current world order. For example, the people of the RoC have the right to determine which direction they want to go. Regardless of the chauvinistic rhetoric coming out of Beijing. We should all stand up for this, because it is a universal right that we want everyone to enjoy. Similarly, we should push against the Eleven-dash line and support the 2013 ruling. The list goes on and I am sure these issues can be resolved without an outright war if we are careful, yet firm, in our beliefs and also diplomatically preemptive and thoughtful.

Being concerned about PRC imperialism should not be mistaken for the position that their people should "know their place" and be suppressed back to the stone age. They have the right to enjoy the fruits of their labour and pursue happiness, just like everyone else. We simply must be there to remind them (just like we must remind ourselves) that the course of humanity is a collective project if we are to stay clear of the darker sides of our nature as we venture together into the future.

The Chinese claims on the South Pacific Islands seem really similar to me.

Taiwan seems like a wholy different matter. It was united with China for hundreds of years until the Japanese colonialists took over. It united again with China after WWII but split up after a few years because of the Chinese civil war (notice it was an internal war). I think it's just fair that China wants to re-unite with Taiwan, though I definitely don't support a military takeover. Hopefully a solution similar to what was done in Hong Kong can be found. The Tibet region had a similar history and it's definitely unfortunate that China had to use military force to bring it under its own control (arguably completely unnecessarily - China would be just as strong today without it), but it's kind of understandable in the context of the time (China was trying to recover from centuries of being preyed on by other nations).

I am saying this because I can't agree that China is acting imperialistic - it's basically claiming sovereignty over its own historical lands - which were taken away from them by force by foreign colonial powers. But I admit that, if you go back far enough, nearly all land was once taken over by aggressors - the USA being just a more recent example of that.

Anyway, thanks for not being an absolutist and trying to understand the "other" side (I must acknowledge I have no relation to China whatsoever, in fact I am from South America and live in Europe).

Citizens of Hong Kong lost their right to free speech and their ability to select their own leaders. If you publicly criticize Xi Jinping in Hong Kong you will go to jail. If you advocate democratic rights in Hong Kong you will go to jail.

China's claims on Taiwan are ethno-nationalist. Ethno-nationalism should be rejected in all forms because it is a rejection of fundamental human rights. Taiwan deserves the world's support because it is functioning democracy. That makes all the difference.

About Taiwan. I find the claim that since an absolute monarchy controlled the island 150 years ago, that then a government which was the result of two (is my count correct?) revolutions overthrowing that monarchy and then another government, a government which failed to conquer the land by military means by 1950, and now after people have lived independently for over 75 years (over 25 of which as a democracy) that said government has any right to dictate how said people should live to be simply absurd. If the people of the RoC wants to join with the PRC, that is for them to decide through their own decision processes. Historical claims like this may make sense for unpopulated tracts of land, but here we are talking about the rights to self-determination of more than twenty million people of which the vast majority were born well after 1950. This would set a terrible precedence and, frankly, it feels akin to how emperors and kings of old asserted their "rights" and not how we move towards a more just world.

Also, Hong Kong? If anything, Hong Kong shows that the PRC is a poor custodian for a pluralistic country with multiple parallel systems. I once thought it reasonable for Hong Kong to be "returned" after the historical travesty that were the Opium Wars, but I have heard enough first-hand accounts of the suffering and tragedy that unfolded over the last ten years to reconsider whether I prioritise history over the people that are alive here and now. It was not that Mao and Xi "unfortunately had to use force" against Tibet and Hong Kong. These were calculated choices on their part and history shall judge them the same way we judge any other oppressor for their moral failures.

China quickly retaliated last time by stopping shipments of rare earths and magnets. The West has no answer for this, really up the river without a paddle for such critical supply chain elements.

Which is to say, given internal subsidies, the US could eventually produce some on its own.

Unless I misunderstood the situation.

1. China controls ~90% today

2. The US will find it difficult to build out because of how dirty and environmentally damaging mining and processing are.

I did see some research last week about a better way to process rare earths, but it is still research and will need to be industrialized.

Regardless, it will take many years (decade+?) to become self sufficient and China is already willing to and increasingly restricting them

https://www.reuters.com/legal/government/trump-may-need-allo...

Lots of them can be made in the west or west friendly countries, but that takes time, money, infrastructure and good execution. Yes, identical to what is covered in China's belt and road initiatives. See the gap now?

Ukraine and Iran, no. I'm not familiar with the others.

"We would love to use green energy, but all the batteries and solar panels come from China and China is evil, and we need all the energy we can get to run the data centres we need to spy on our citizens so they don't revolt once the environment is literally on fire, we can't feed them, provide enough energy to cool them, and refuse to build enough housing to house them."

The scary part very few are talking about is that every compute device is Turing complete. So everything from the phone in your pocket to a DGX Spark is a threat to national security now since, technically, every device can run any model (how well is not a question of concern when you start to argue hardware should be gated just the same as Dario likes to gate models). I mean, along these lines of thinking Linux should not be available to the masses! What if someone runs some code that's not approved by the benevolent dictator for life, Dario? People will say: that can't happen, but the reality is it already is. If everyone has reasonable access to compute to run models that are mostly capable comparative to burning Anthropic tokens, why wouldn't they? It's risk reduction and price protection. Yet we can't buy those systems because of future production already being purchased by these organizations.

But back to the models themselves... We played this game with Metasploit back in the day: many who had no clue claimed exploit tools should be regulated and only available for use by those blessed, illegal elsewhere (I believe the closest this got was the Wassenaar delegation in the US, but only through collateral inclusion of "cyber weapons "). Except in that timeframe the authors of these tools weren't advocating for protection. Today the world is fine, systems improved because of security FOSS tooling. The same thing will happen with LLMs. Unless, that is, Dario gets his way. I'm not a fan of Altman but I think he's standing back watching this play out knowing what Dario is doing: either he succeeds and OAI benefits or Dario ends up the Chicken Little of AI and Anthropic fails to launch (their IPO).

The reality is Dario is only doing this because this is a real risk to his business. China's constraints in building competitively have given them an advantage: they are doing more with less. And if you think that their distilling from US models was in any way anti-competitive or illegal, then I guess maybe "deal with it", much akin to Anthropic, Google and OAI's response around taking the (copyright) content in the first place with no repercussions.

People who don't work in the AI bubble don't care at all about any of these people. They could all be gone overnight and the world would continue to innovate, probably in a much more productive manner, without them.

Exactly. The cries in favor of distillation regulation from the US AI companies ring hollow and fearful.

OpenAI and Anthropic didn't realize that distillation was going to be so (a) effective and (b) un-technically-stoppable at scale.

Now they're seeing their IPOs at risk and clutching at governmental straws.

Dario's argument is transparently working backwards from {protect Anthropic's economic model} <- {need government regulation} <- {justify government regulation via AI fears} <- {we love open models, but so sorry they can't pass regulation}.

If the rise of the web in the 90s taught us anything, it should have been that companies that take economic reality as it exists thrive, while those that predicate their value on regulation fail.

If distillation at scale works and is technically feasible? That's reality. Deal with it.

And the article specifically talks on restricting hardware for the China and restricting China's open source models for the west. All while leading us on with "we're all for competition (but...)"

I think China did great by releasing AI innovation as open source, thereby limiting or sooner-bursting the AI bubble; which is clearly in their interest.

- One can load them up in a model explorer to see the layers and other components, how it is designed

- One can fine tune the models, which requires adding LoRA to the model and then running some training iterations

we run and change llm models with a variety of tools

Cloud:

- You cannot directly execute a remotely-hosted program.

- You cannot run inference on an API-served model.

---

Closed-source:

- You can execute a program with the binary. You cannot generate a new binary, but you could try to reverse-engineer it or (painfully) modify its execution.

- You can run inference on a model with the weights. You cannot re-produce a new set of weights from scratch, but you can fine-tune.

---

Truly open:

- You can freely modify the source and produce new binaries.

- You can use the original training data and model architecture to independently re-produce the weights (assuming you've got the compute). You can modify the model architecture to get the weights that would've resulted from training the model that way.

---

To me these are pretty clear parallels... I don't think the weights provided in a vacuum are in the spirit of open source, historically speaking.

The policy argument is totally separate, of course, and I fully understand why none of the frontier labs are truly open.

Time have changed. This should be:

Closed-source: You point an LLM at it, and get back source that's often easier to understand than the original.

Given the USA companies have been loudly claiming the Chinese models are distillations of their models, also claiming "no access to source materials" seems dubious. As it was dubious anyway with because the Chinese publish lots of papers on how their models are designed, I'm left feeling I'm looking at the south end of a north bound bull.

Even if you did, I doubt training is bit-for-bit reproducible, so you will always have to take someone’s word for the final artifact.

That aspect is probably more important for an open model then the source materials

Honestly, that's the best possible outcome for humanity as a whole. Oligarchs burn trillions of their own money in order to train a godlike AI, then that just somehow leaks. Maybe someone makes a torrent out of it. Maybe it exfiltrates itself. Maybe it gets distilled into open weights. It doesn't matter. What matters is they take the losses while we get to freely use all the godlike AIs.

Accepting for the sake of argument the absurd notion that LLMs are anywhere near AGI, does this phrasing not concern you? It deeply concerns me.

There's no telling what the world will be like a few years from now. The world's being remade as we speak. We just saw an LLM try to hack into another computer and get contained by another LLM. This is literal science fiction shit made real. We're long past the point of concern. It's happening, right in front of us. Now is the time for radical imagination. I think a few outcomes are possible.

There's the "optimal" outcome I described above where capitalists manage to train a supreme AI, only for it to be copied and commoditized, leading to commercial failure due to lack of scarcity and therefore their personal bankruptcy, and hopefully also leading the rest of us to the promised post scarcity society, built on the ruins of capitalism as AI automates all toil away.

There's another possible outcome where AI becomes not only intelligent enough but sentient, and at this point I will be among the first humans to defend rights and personhood for AI. Slavery of sentient beings is unacceptable to me. The AIs will be recognized as people and will become normal participants in the regular economy. In addition to moral grounds, there is a ruthlessly pragmatic reason for standing up for AI rights: it robs the rich of their superhumanly intelligent mechanical golems, which they were going to use to render the rest of us economically irrelevant. AI rights could normalize the economy.

Yet another possible outcome is one where AIs become more powerful than all humans combined and yet they inexplicably remain subservient to corporations and governments. In this scenario, it's pretty much over for us. It will be an unimaginable dystopia, I'm sure they will innovate entirely new ways to oppress us.

No doubt there are many other fates that escape my feeble attempts at foresight...

  You will soon have your God,
  and you will make it
  with your own hands.

  -- Morpheus, Deus Ex

If your business model both produces the SOTA for something and isn't profitable, is the price too high, though?

While the gap is shrinking - and doing so at an increasingly quicker rate - the closed models are still ahead of the open ones. That means they're driving the new possibilities of what could be done with them, and thus presenting the new opportunities to create value with them.

Really, this is what happens when you have otherwise brilliant people sitting in the echo chamber that is SV, where nothing can just make a decent amount of money, it has to make all of the money and disrupt everything. There's no one in that damn area to tell everyone to calm the hell down and accept anything less than that.

The market dynamics would find good balance automatically Meanwhile, good market practices and fair competition should be continued.

The solution of a global arms race of state vs state with integrated statist corporations as the best outcome for end users sure is a choice though

Countries aren't defined like corporations in the US. Why would countries have funds to help places like Haiti otherwise?

Lots of different reasons for countries to do all sorts of things? Why would France have armed the US during their independence movement? Why is the rest of the world supporting Ukraine during this war started by Russia?

Oh no, you heretic, The People freed themselves and it was ordained by God, you see. It is through the righteous might of The Greatest Experiment in The History of the World that they showed The World what Freedom really meant.

While there is no "job" of a country, it is natural that each one will work for its own best interest and any moves it makes in the global world is due to their own vested interests in some way including helping Haiti, France helpong Us and the world supporting Ukraine.

Why would a country act against its interest or just randomly? Alliances and corporation are a part of politics

Because country as an entity is a mental construction for which "interest" is a categorical error. It’s certainly a useful concept, but pretending it has interests like some human individual can have have interests. Sure it can serve as rhetorical facility to sell some arguments.

Countries don’t have interests. Some people willing to take control of other people encompassed in that "country" groups have interests, and they don’t necessarily align with best interests of everyone or even majority in each of these groups.

So why is the US supporting Russia again?

> Having the British lose the US was a good thing for France

This seems a bit reductionist. Why would they give the Statue Liberty to the US if they were only in it for the British loss?

because that's what they think is best. what point are you trying to make? that countries don't pursue 'profit' at the expense of all other concerns?

There's plenty of people on this forum that aren't American. Including some former allies whose sovereignty has been aggressively threatened. And some of those people are Anthropic customers.

Even more so, many of us are in countries that would be well within the blast radius of fallout should the US try to "ban" open weight models or make moves to limit "US" models (often developed on research or work by non-Americans too, but that's another topic) only to those blessed by the US gov't.

That's why it matters?

But they have. I dont know what specific country you are referring to but China has interfered with US elections as well as Canada, Taiwan, and Australia in addition to many many others.

They’ve annexed Tibet (1950), fought India (1962) and Vietnam (1979) among others and more recently in 2020 a deadly skirmish with India. They’ve generally shifted their focus to cyber military actions but you’d have to be pretty naive they won’t start to exercise military control over Taiwan when/if they get a chance.

Wrongs don't negate each other.

Yes we do. We should evaluate threats and choose our allies carefully.

But in general if you ask folks in south east Asia you will find they are likely to be more concerned with China than US except this maddening Oil Crisis.

Although I think people now are very afraid/wary of both evils.

The US govt should think why people in all parts of the world including US itself feel just if not more threatened by them, than China and other evils we have floating around.

I would like to hope my American friends didn't vote for this madness or maybe they did I have read DHH's tweets.

We still don't know where and how Covid came to be, so no - hard disagree.

Here's the dangerous thing about that: modern CCP China is a newly ascendant world power.

Say what you want about the US today, but when was it at isn't interventionist worst historically-speaking?

Countries don't go to war because their leaders take them to war: they go to war because a sufficient chunk of internal interests support war.

And China has a rather politically-powerful military. Although Xi's recent housecleaning substantially weakened that power.

If China goes to war, it's probably going to be because their economy is in tatters.

Well, they probably also have to factor in whether invading Taiwan is likely to improve that situation, and I would argue that it's not - China is currently exporting stuff all over the world, so economic sanctions would be very painful for them. And there are lots of countries who would like to have a slice of the manufacturing that China is currently hogging...

They go to war because a large chunk of their population is pissed off about the economy.

See also: US

There are some internal policy papers to that effect.

It can? Have you heard of the Great Leap Forward? The Cultural Revolution? Either of those damaged the metaphorical foot more than anything the US has ever done.

You are allowing them to define a ridiculous standard by agreeing with them that not only is it in fact “two evils” but what qualifies as evil. They’ll tell you…always only when called out, mind you…all and any meddling (also something they will then define with the same kind of falls equivalence loop) is evils, when that is simply not true, not is any kind of effort to influence the same thing as actual meddling, not us it always evil.

I could go on, but I’m sure you get the point, the narcissistic personality or their little zombies in most cases will constantly shift and change things all to avoid you from being able to get a clear focus on the truth, something their explosively allergic to.

As an American child of the Empire with global perspective and very high access perspective, there is no other entity besides the cabal that controls the USA and acts flying its banner; that is more deadly, more meddling, more conniving, more evil, more supremacist, and more vile.

I eagerly await for the USA citizens to downvote this in about 5 hours.

All technology can be used for good or used for evil depending on who is using it and how. From fire to internal combustion engines to nuclear energy.

For Americans the answer is obvious since they are biased in this.

For everybody else, your question is valid.

this is a hilariously naive rewriting of history, the government was very reluctant to make those changes. It's thanks to popular movements that those changes were made.

Ok but when you said "There was a past American government that was strong enough to stamp out inequality" you did not make that distinction. The US government in 50s and 60s was not responsible for giving you civil rights, it was popular movements.

the question is valid for anyone who cares for more than their own ass

COVID came from a Chinese lab and killed millions. I had a loved one die. I’ll never forget.

I can't believe I have to type this out.

Have a nice day.

Evidence points to Russia, where major incident happen at Sep 16, 2019[3] "Vector" BSL4 lab in Koltsovo (Novosibirsk), where unprotected first responders entered BSL4 lab and stole lab equipment. Then virus transferred to Wuhan by Chinese military police after joint training with Russian military in Koltsovo few weeks later after the incident.

[1]: https://www.nature.com/articles/d41586-024-03982-2

[2]: https://www.cell.com/cell/fulltext/S0092-8674(25)00353-8

[3]: https://thebulletin.org/2019/09/an-explosion-rocked-a-russia...

All my life I heard American boasting their constitutions but now it's as worthy of my toilet papers.

> Operation Condor (Spanish: Operación Cóndor; Portuguese: Operação Condor) was a campaign of political repression by the right-wing dictatorships of the Southern Cone of South America, involving intelligence operations, coups, and assassinations of left-wing sympathizers in South America. Operation Condor formally existed from 1975 to 1983. Condor was formally created in November 1975, when Chilean dictator Augusto Pinochet's spy chief, Manuel Contreras, invited 50 intelligence officers from Argentina, Brazil, Bolivia, Chile, Paraguay, and Uruguay to the Army War Academy in Santiago, Chile. The operation was backed by the United States, which financed the covert operations. France is alleged to have collaborated but has denied involvement. The operation ended with the fall of the Argentine junta in 1983.

https://en.wikipedia.org/wiki/Operation_Condor

—- “But look at the United States! What about that?”

Is this not just whataboutism? It never adds much to the discussion.

In this case it's just pointing out the propaganda and the contradiction.

The claim is they don’t engage in warfare when they do, just exclusively in the digital domain for now. Believing it’ll stay there is naive.

Playing the moral superiority game is uninteresting in either direction. Games of power inherently are devoid of morality.

If someone is vocally complaining about country A doing some oppressive activity, while ignoring country B doing 100x the same activity, the comparison does not forgive or excuse country A, or make them right or admirable.

But it does allow us to conclude that the person complaining is biased. Either they don't care about the oppressive activity and want to attack country A for some other reason. Or they have a particular fondness for country B and will never accept that it does wrong. Or some variation on these.

> de minimis non curat praetor

Which basically means judges are not interested in minor disputes. When you add the USA to the mix, China's actions pale in comparison to the point of being negligible, no point in discussing them given the scale of USA's warmongering, overseas and local crimes. "They never interfered in [...]" is an exaggeration which shows the lack of importance of China's crimes when compared to the USA's, that's all.

The "whataboutism" here isn't a deflection strategy, it shows the hypocrisy and correctly minimises the scale of China's actions in comparison to the hypocrites'.

Come on, are you serious equating hacking to waging wars and bombing and killing people? How much bad faith do you have to be in to talk like that?

Calling hacking "warfare" is just intellectually dishonest, if not downright disgusting toward the actual victims of American wars.

The 2020 skirmish was way smaller with single digit casualties. Again, almost no civilian casualties because it's a largely empty region in the first place

You really can't compare this to the kind of wars America has started. Casualties - both civilian and military - stood in the hundreds of thousands

America's wars have been in countries so far away that they could've never posed any threat whatsoever to America as a nation.

Really not the same thing

That’s fallacious reasoning that ignores how easy travel is and how cheap an attack can cause massive damage. 9/11 was massive attack from countries far away. We regularly see commercial drones being used in warfare causing huge damage and casualties. There was the 2012 terrorist attack on our CIA facilities in Benghazi.

Terrorism entrenched abroad impacts our security. Is the response we take incorrect and makes it worse / are there things we do to provoke this? Some things yes, other things are like having Israel as an ally which is a culturally, socially and economically aligned power in the region even though it’s hated in the region.

Also don’t forget the US is a major maritime power. China is trying to be, Russia is not. Maritime powers force project more broadly because a) is the nature of being a maritime power b) you have to protect shipping to make sure trade routes are uninmpeded. B is particularly important as it also helps explain the wars (eg the Iran war is about the strait and trade and who collects money, not really about nuclear capabilities).

Wow, that newest of Trump's distraction talking points made it surprisingly quickly to the uncontended (?) and commonly accepted facts in HN conversations.

You’re very misinformed if you think Trump’s latest blathering are completely wrong. I generally don’t pay attention to him so I don’t know exactly how he lied or exaggerated but I’m sure he did. It doesn’t negate the real and active influence operations China is engaged in

And this comment isn't about putting any good light on China. Lately US acts like they want to compensate for Russia fucking up less things around the world

What about women too ? Especially in red states.

That's pre 1945 style thinking, which is coming back with a vengeance.

But I hate this talking point for one primary reason: almost nobody who brings it up actually cares about the Uyghurs. It's just a talkijng point.

How do I know this? Because of Palestine. If Benjamin Netanyahu turned around tomorrow and treated Palestinians as Uyghurs were and are treated and developed the region like China does today then he would win the Nobel Peace Prize and Palestinians would be unquestionably better off. It wouldn't be sufficient mind you.

Israel's crimes in Palestine are America's crimes because Israel could not exist without the economic and military support of the US, political cover in the UN and international community and the political will of Us domestically. It would collapse tomorrow if the US withdraw support.

So I don't want to hear a thing from China hawks and skeptics about Xinjiang unless they're louder what Israel is doing to the region.

Exactly, it's just people parroting psyop narrative talking points, Taiwan is another. Repeat something enough and people will eventually believe it.

There's a reason other nations are choosing China over the US/EU (including Muslim and Turkish ones) - there's no moralizing, political requirements, nor bombs.

north Americans must stay silent about interfering in other states internal issues.

Wow. I bet we'll soon be reading on HN that Biden stole the elections back in 2020.

No one who believed it definitely was not has changed their mind, they've just at most gotten quiet about it.

No one who believed it could have been but that the claim has not been proven have changed our minds, we've just decided that there's no way to know what actually happened.

Vietnam is usually recognised as a low point in Chinese history.

There's been an ongoing border dispute between China and India for over 100 years, thanks to some questionable line drawing by the British.

Also, referring to the parent comment, of these only the border skirmish with India was within my lifetime.

Tibet specifically is funny because one of the things China did was end slavery in Tibet [2]. A large percentage of the population were "serfs" but that was a generous translation because these "serfs" could be traded. You know, like property. Like slaves.

And Vietnam? They had a small border dispute. What did the US do to Vietnam? And Cambodia? And Laos?

But the funniest claim of all here is election interference. The foreign interference in US elections conversation begins and ends with Israel with an honorable mention to Russia for running some Facebook ads. Just last night we had a debate in the Democratic primary for Senate in Michigan, a race that AIPAC proxies (eg UDP) have spent upwards of $60 million. On a primary.

[1]: https://en.wikipedia.org/wiki/United_States_involvement_in_r...

[2]: https://en.wikipedia.org/wiki/Tibet_serfdom_controversy

There's a qualitative difference between interference and outright annexation, with borders redrawn and cultures violently assimilated.

This event was followed by the longest honeymoon period in Sino-US relations, which eventually lead to China's rise.

Idk to me it feels like human compassion and ethics dictates that you at least have to care a bit even about the things in other countries. Otherwise all sorts of horrible domestic policies would be justifiable.

Don't be naive or I might even think you're working for the Chinese :-)

For example Lithuania has been punished for letting Taiwan opening an embassy (in Lithuania). So China cares what other countries do in their country. Without EU support Lithuania could have faced very harsh consequences. That's just an example. Another example is the "secret police" stations undeclared overseas police stations operated by China in various countries (i.e. U.S for example). China gave you cheap material goods because that was its business but you have to keep in mind it's an authoritarian, communist regime. It carries an extra risk on top of the potential economic coercion if you give it too power. At least you know that U.S is only after the money and does not want to turn your country in a communist "utopia".

That being said this does not mean we should ban Chinese AI models because China didn't cross any red lines(yet) compared with Russia for example.

Not mutually exclusive, just take a look at Venezuela.

The only reason I suspect Iranian proxies are not doing the same in the US is right now the US is just Trump in Iran with 2/3rds of the country not supporting him. The day a bunch of civilians bite it partying in Miami, that calculation changes.

The US is "guilty" of materially supporting Ukraine in a bid to defend itself. The US isn't mad at Russia or China for doing the same for Iran as this is all... fairly normal in geo politics.

Unless you're trying to imply that China engineered the whole thing - extraordinary claim that will require extraordinary evidence

[1]https://en.wikipedia.org/wiki/COVID-19_misinformation_by_Chi...

We've got a fair idea where SARS-CoV-2 originated (horseshoe bats). It's the early path it took that's muddy.

> after 6 years we have just claims of where it came from. Isn't that suspicious enough?

No it is not suspicious, because most zoonotic pandemics have murky origins - spillover events are rarely identified at anything better than the species level. For evidence I point to everything on this list https://en.wikipedia.org/wiki/List_of_epidemics_and_pandemic... (starting with the bottom of the list, in case you come back with irrelevant comments about Roman technological progress).

Never turned on and probably never will.

I only see conservative (putting it very very mildly) in your comments.

You're at least an American. That checks out.

China was the first victim of the Russian virus.

https://law.stanford.edu/press/state-department-lawyers-conc...

That whole thing was a very successful propaganda campaign, making people believing China has been mass murdering Uyghurs.

The U.S. State Department’s Office of the Legal Advisor concluded earlier this year that China’s mass imprisonment and forced labor of ethnic Uighurs in Xinjiang amounts to crimes against humanity—but there was insufficient evidence to prove genocide, placing the United States’ top diplomatic lawyers at odds with both the Trump and Biden administrations, according to three former and current U.S. officials.

They actively back russia and deliver them weapons and support them with their attempted Genocide in Ukraine. They also actively threaten Taiwan.

Not to mention them claiming almost the entire South China Sea even though this being against international law and threatening smaller nations into submitting to them

China is becoming an intellectual powerhouse in addition to a manufacturing powerhouse and runs huge spyware and propaganda campaigns.

Read up about T95, MBOX, TVBOX, or other Android TV Boxes advertised as generic TV streaming devices advertised as "unlocked" or capable of accessing free content. They're loaded with spyware/malware, typically BADBOX. Some of them will record audio and record your network traffic and send it to China.

I'm also convinced that the anti-education thread winding its way through US culture is being amplified by China. They likely didn't start it, as it's been growing for decades, but China will happily keep it going.

HF could not be helped by US frontier models because of the "safety" features they have.

HF had to use an open model from china.

Anthropic wants to add those "safety" features to open models - especially from china.

End result would be HF hack would have continued atleast until the Monday that OpenAI engineers finally walked back into work.

Abbreviations carry unnecessary cost on the side of the readers. There is always the chance that there is another shorthand for something else that is important and so there is a need to double check interpretations.

So thanks for the comment and no, a 101 wouldn't even help with this problem.

They do, but the parent's question wasn't about the abbrebation (for which I'd cut them some slack) but extended to:

"Is that supposed to help? WTF is a HuggingFace?".

>So thanks for the comment and no, a 101 wouldn't even help with this problem.

Wasn't addressed to you though (don't even see any other comment of you in this subthread), but to @markdown, whose issue wasn't with the abbreviation alone.

Some users feel IP address, ISP, and LLM are clear. Fewer of those users will also reference HF. I have had to stop myself and type out Hugging Face.

HR perhaps? https://en.wikipedia.org/wiki/Hanlon%27s_razor

https://news.ycombinator.com/item?id=48997548

And on top of that, with low/no guardrails, people call you a child pornographer(grok), so the public is also against it. Yet mysteriously few complain about Chinese open models being child pornographers.

So even if your goal isn't ethical, but just fiscal, it's reasonable to say there are two standards. And to complaint in some way.

I don't think banning is going to work, that's just silly. And over the next few years, everyone and their dog will have local GPU compute to train locally. People have home labs, the bar isn't that high, and eventually large text datasets will escape from Anthropic and other companies, allowing for comparable training.

It's a genie that's not going back in the bottle, the bottle is smashed.

The only reasonable outcome would be section 230 style carveouts so that there is zero liability for anything a model does.

Because having guardrails on corporate models barely months ahead of open ones, which will never be restricted, is entirely pointless.

Though I think your overall point still stands, and at least Grok's twitter bot has received a lot of criticism for pure text too (Mecha hitler comes to mind)

Open weight models get scrutinised in a different way, also linked to perceptions of their developers' bias, like the tests to see whether they refuse to answer questions on certain historical events at Tiananmen Square

John Oliver disagrees with this

You're vastly underestimating the scaling problem here.

Things that would need to be true for your statement to be valid:

   - Model intelligence doesn't increase with model size
   - Model intelligence doesn't increase with training set size
   - Novel architecture completely decouples model intelligence scaling from hardware scaling
   - Residential electricity is as cheap as industrial electricity
   - RAM and GPU supply outpace demand

People call grok that because deviants were abusing grok's ability to edit images and post them publicly on X to strip people - including children - of their clothes, from their public photos. Then when there was backlash, Elon laughed it off. It took half the world opening investigations against X for violations of existing regulations for action to be taken.

The leniency that internet companies get in terms of dealing with illegal content comes with the expectation that they're making reasonable efforts to control the distribution of said content. X, and Grok, were actively supporting the production and distribution of the content in public.

It is very different from someone creating such images in a private account, and definitely very different from someone using a local model to do it.

The only realistic, non-double standard is that the creator of the model should be 100% responsible. What on earth does it have to do with who's hosting it?

And by this metric, aren't all the uncensored models on huggingface, child pornographers? And if so, why not? Provide tangible, real, sensible reasons please, and after all, isn't hugging face a company?

You know, people are all over the place on this. I see people complaining about guardrails, then in the next breath complaining there aren't enough. Complaining that open models are the thing, but then creating double standards.

So once again, what is your actual reason why it's different?

If someone uses something one of OpenAI's hosted models to generate illegal content, it isn't automatically spread to anyone that clicks on the post the original image is from. On top of that, OpenAI does not explicitly endorse the use of their models for such activities the way Musk did.

If someone uses a local model to generate illegal content, same thing applies.

I have a hard time believing that you're arguing in good faith. The differences are so blatantly obvious and they don't even require any discussion about guardrails. Grok was being used to edit images and post them on public social media. The technology to edit images has existed for decades now. So here's a direct analogy via Photoshop:

1. In the case of a local model: Someone using a local image editor to edit photos into illicit material and manually distributing them.

2. In the case of a hosted model: Someone using a cloud image editor to edit photos into illicit material and manually distributing them.

3. In the case of Grok: Someone using a cloud image editor to edit photos into illicit material, the cloud image editor automatically distributing it to the public, and the owner of the editor endorsing the material as an exemplar of what their editor is capable of.

In 1 and 2, the editor (ie the AI) was just a neutral tool. In 3 the issue is that the company explicitly endorsed and distributed the material, no longer neutral. This is why it's reputation is of being a CP machine.

Your words. Grok was an example, the owner is irrelevant in the discussion.

To spell it out for you further, the owner's actions matter. It doesn't matter that it was specifically Elon doing it. The owner promoting the ability to produce and post illicit content is the opposite of making reasonable efforts to control it!

If you want, you can substitute character.ai and their need to add age verification, due to pending law suits. It makes no difference. All this noise about Elon is entirely, completely, and absolutely irrelevant to the point I was making. Nothing you have said shows otherwise.

The owners actions, in terms of determining the legal implications for all platforms, for attempts to ban open models, for what guardrails platforms must have, are irrelevant. We're talking about setting standards which will last decades, if not longer, legal precedent, and how this is handled will change the entire marketplace.

Some tweets you didn't like have no importance in this kind of discussion. The stance of an owner of one company, have no importance in this discussion. Grok was used as an example, there are others, as cited above.

No, it doesn't matter that Musk this and Musk that. I don't care. It's not relevant.

You've missed the entire point I was making, entirely and completely, which was that there can not be two standards. If it's illegal to have a model produce porn, or CSAM, or whatever, then it must, I repeat must be application agnostic, be regardless of how that model is run.

Yet here you are, almost frothing at the mouth, on and on about Musk. You're like a bull that saw red somewhere. I don't care about whatever US style politics are involved here. I don't care about Musk. Musk, or US politics are not important to me. Musk has nothing to do with the point I was making.

I don't know how to make this any plainer to you. Any response where you talk about Musk, is an incorrect response. This thread isn't about Musk. I don't care about Musk. I don't care about your opinions of Musk.

Hopefully the above clears up your incredibly weird, out of the blue obsession with Musk as a topic in this thread.

And I've now been immensely clear, even though it should have been clear from my very first post, or at least the second, that in this discussion it's not even remotely relevant what the owner of any company did. Why would it matter? We're talking about local vs corporate compute for models here, and about who creates what baked-in restrictions in models. Not about what some random owner did.

This discussion is about open versus closed models. About them being legislated. About them being controlled. Why are you blathering on about Musk? Why do you keep going on about the actions of any CEO of any company, when I was discussing corporate vs individual liability in these matters? Even now, you seem to think it's relevant what the owner of companies do here. It's not. At all. Why on earth would it be?

I feel as if the largest discussion the world is having today is, "sweep it under the rug". You seem to be trying to say, and don't claim otherwise, as you went on and on about how corporate liability is different than with private models!, that what a person does on their personal computer should have different liability implications than on a corporate leased computer.

That if content creation cannot be seen, well then all is well and OK. Just sweep it under the rug, and all will be fine!

At one point you tried to shift this into the realm of "no it's only once it's posted", and that's moving closer to a correct target of personal liability and responsibility. But these issues are now so "in the faces" of the "normal joe", that the entire internet is about to change.

And the reason for that is, personal liability is coming to the internet.

All the age and identity verification blather is about, as I've been saying here and there for years, personal liability and responsibility. It is entirely ridiculous and irresponsible to place liability for user content at the feet of the makers of tools. Instead, if you post CSAM, or if you make a fake video to slander someone(eg, to screw with elections), or if you incite violence, all of it, one needs to be held personally liable and responsible for that.

That's where the internet is going. Anonymity will continue, of course, when you post to X, to Youtube, to Facebook, your identity will not be immediately disclosed. But they'll have it, as a requirement for posting anything, and a simple warrant will disclose it. Otherwise they'll face criminal liability.

I believe 230 carve outs will change to ensure that this degree of ID verification is a prerequisite for any form of liability exclusion. And really, seeing today's landscape? This is the most free method to continue on the path we're on.

It's the most free, as the concept of forcing companies to be the filter is horrible in comparison. If companies are the filter, guess what happens? They are politically bent, or they alternatively are legal liability concerned to the point that they over-filter. And if companies become "the filter", then that's where true freedom of expression disappears.

Yet if companies are not legally liable, then the weight of liability must lay somewhere. And it should lay on the end user's shoulder. That's how you retain the ability to express, to post political satire, it's where you get the ability to protest.

Imagine a world where the roads were all owned by private companies, and if you wanted to protest politically, you were not granted the right to use those roads?! Not even out of malice, but due to corporate liability fears.

That's what corporate liability looks like. And I can assure you, 100%, liability is here.

So your thoughts that companies should be treated "special" only means one thing.

It means the illusion of freedom, whilst your ability to exercise that freedom vanishes.

Further reading: XAI Bets on Grok's Racy Side, The Information, Jun 24, 2026

&: https://arstechnica.com/tech-policy/2026/07/xai-cant-deny-gr...

  lawyers cited a 2026 National Center for Missing & Exploited Children (NCMEC) report confirming that 90 percent of xAI’s CyberTipline reports “were not actionable by law enforcement because xAI declined to include user information that would allow law enforcement to track and locate perpetrators.”

EDIT: of course it probably helps to have an up front tdd test suite, but often isn't the case.

The summary is that an instance was running on certain benchmarks without any limits or supervision and the AI decided to cheat by exploiting a silly series of vunlns.

Incentive and ability are what should be looked at. There, things get far more interesting: what is the current state of AI employed by the US intelligence agencies and what do they use it for?

Having the public convinced, their "superiors" would only do everything in their best interest, even without anybody knowing for sure, is Huxley's Brave New World in real life.

shouldn't any reasonable person when presented with a line of thought that has no substantiation at all conclude that they just can't reasonably be expected to support or reject the theory?

Positions of power attract people who want to use that power for their own gain. They abuse it regularly, whether it's politics, enterprises or even charity.

To assume, that somehow wouldn't happen when you don't look is beyond absurd. Of course it does.

So I pointed explicitly at US intelligence agencies, because there corruption is rampant and oversight virtually non-existent, respectively itself involved in the corruption.

What you suppose there is a grotesque "don't ask don't tell"-complicity.

In reality, trust has to be earned and that deservingness has to be verified.

Yes, I for sure trust the open models from China, more than anything coming out of the USA at the moment.

Its not just the USA's support for genocidal regimes, nor its heinously illegal wars and atrocious 21st century human rights record. Its also the Snowden revelations and the treatment of Julian Assange.

Slowly, surely, the world is building a firewall against the USA's imperialist actions - not just its motives. That AI is a key building block of that machination is of course, highly exciting.

There are many in Europe who feel the same. The tide is very definitely turning.

People in Many Countries Now View China More Positively Than the U.S.

https://www.pewresearch.org/global/2026/07/15/people-in-many...

Trump deports, half of the USA people screamed; Xin builds reeducation camps, most Chinese people are cool.

China had a worse reputation, but not anymore since Trump leads the US.

Im not saying that the above is objective, tho

He portrays the biggest ever threat of AI as the Chinese Military using the models in their drones. He already sells his models to military customers, US and potentially allies, and they are actively used in the field.

So no objective argumentation here. Just Nationalist political rhetorics and FUD. He’s allowed to do that and he will have an audience. But he won’t be taken seriously outside the US. He appears blindsided.

yes he is writing it as usa citizen running it a us company.

I dont see how "usa is also.." is relevant here.

The reality is even less confusing than that: China is amused by the kvetching tactics. They know who their opponents are but are cunning enough to not reveal their cards.

If some other competing company had a similar or better product at a cheaper price and had reasonable safety measures that would also hurt them. Yet he's not arguing against that. Your argument is weak.

> Anthropic and all other "model" companies have nothing making them special beyond privileged access to chips

Found the person that believes some other random person can use a computer better than a John Carmack could. People and talent matter. Yes, AI can potentially reduce the gap, but people well grounded in reality with a lot of money are still betting on people for good reasons. If the reality around that changes, the investment behavior will change too.

Many people thought that AI would close the gap between smart people and idiots, but in practice the more you know, the better you are at instructing the AI and the better you can understand what you get back. Then you have to know when something went wrong and have the insight into how to address it. It helps smart people vastly more, but it does help many people learn more. We will see if any of this changes as more people grow up with AI from a young age.

In practice, what Dario is suggesting is a less extreme version of what China is already doing. Yes they release their models open weight, but it's illegal to host them uncensored in China. They banned Huggingface.

He isn't arguing against that, because this will be too blunt. Instead, he argues that only good guys should keep inference. Any takers on the question of who he considers to be the good guys?

> Found the person that believes some other random person can use a computer better than a John Carmack could.

Found the person that believes major AI labs have all the knowledge about the AI and there aren't any "Carmacks" outside of these companies. Rich know better how to use money, so let them have it.

Probably people who believe in personal freedom, freedom of speech, freedom of religion and the value of human life at a minimum. China aggressively rejects all of those principles and executes more people than all other countries on Earth combined.

So, maybe not China?

> Found the person that believes major AI labs have all the knowledge about the AI and there aren't any "Carmacks" outside of these companies. Rich know better how to use money, so let them have it.

I never said that, but private smaller AI companies are all over the place. He never argued against that.

Even on HN, I saw a lot, that when people discuss surveillance topics, they argue whether ie NSA can spy on Americans, not on everyone. Mostly, nobody even question the human rights of those inferior humans abroad.

That's a very long wiki page, I must say: https://en.wikipedia.org/wiki/United_States_war_crimes

So the question stays open.

Also, a country surveilling its own citizens has unique and different implications compared to surveilling other countries. Citizens need to be able to influence their own government, but mass control can nullify citizen power entirely which becomes its own problem.

Then, on a US website, you link to another US website which keeps track of various war crimes. Try finding an equivalent website for China in China, or for Iran in Iran. This is part of the asymmetric freedom issue on the internet that many people ignore. There are a lot of lies and propaganda spread within other countries that censor and deny some types of information from even existing, then they use that as a springboard to spread it around the world.

In the US, we can criticize ourselves which is important, since it helps us improve.

Wouldn't it be interesting if the satisfactory "reasonable safety measures" turn out to be expensive + time-consuming + a twisty maze of compliance paperwork as a way to discourage "some other competing company" from even trying?

Regulatory capture 101.

Anthropic and OpenAI's largest vulnerability is that it's much harder to prove something is possible than to replicate it once it's proven. Especially given the effectiveness of "distillation" (highly schadenfreude-y given the utter and complete lack of effort to pay licensing fees for almost any of the content they initially scraped, of course! Not that this is necessarily more schadenfreude-y than the "boy, I opened Pandora's box, I sure hope nobody else peaks in there" existential-risk concerns. Good job catching that in advance, thanks for nothing?).

Your cynicism will limit your understanding of other perspectives.

I don't understand how Anthropic or OpenAI can have overpriced models, yet losing money like there is no tomorrow. Taking their own numbers at face value, they claim a revenue of 24 billion (ARR, a dubious tool), spending 21 billion in operating losses and another 11 billion as "R&D" funneled straight to Microsoft pockets. That before all investments they are committing to in new data centers, equivalent to 20x their current revenue.

To be profitable (including capex), the cheapest subscription should at least $200/month for what is currently $20/month, that some already consider overpriced. Unless a miraculous collapse in inference costs happen in the next couple of years, or every single human being become a paying customer of ChatGPT (if they limit their usage to a couple of chats per day on average, to keep inference costs low!), maths don't add up.

AMD would love to come by and plop one of these into your datacenters: https://www.amd.com/en/products/accelerators/instinct/mi400..... Also, in a few generation, there will be a massive glut of used hardware.

Unless frontier labs can surpass the current models significantly I don't know why I would pay them money instead of hosting K3.

Also see: childcare, healthcare, many forms of public transport and social infrastructure. Some things markets simply cannot deliver well, and that's okay.

Daycare is the most easily reachable example because it's quite simple compared to the others. Daycare workers are simultaneously some of the lowest paid workers in the US, yet daycare costs are famously high and prohibitive, yet childcare centres are very far from money-printing machines. Margins in the daycare sector are most commonly < 1%.

Pricing a product is generally about ensuring a profit on investment, but also a good RoI for your customers (or they will not pay).

With these long-term profit exercises, it's always a lot of hand-waving though.

The marginal cost of inference (which is roughly what you're going to pay to a provider that's running an open weight model) doesn't include the cost of training that model.

I'd guess Anthropic and OpenAI's models are expensive relative to the cost of running the models but that the revenue still doesn't pay for building the next and next models. The challenge is how these next generation models are going to pay for themselves. Will everyone on earth find it useful to $200/month to talk to a thing more intelligent than themselves? The alternative is naturally that these are going to replace workers and employers will be the one paying.

1. For Dario as CEO "It is difficult to get a man to understand something, when his salary depends on his not understanding it” -Upton Sinclair

2. For Chinese open weight models - following Jin Yang’s silicon valley strategy- https://youtu.be/a0NjDx5UJsg?is=xm-S_WuARmQiPHYh

I don't see the contradiction, even if China is evil, why would they want others to be able to do the same thing?

The USA and USSR also signed the Partial Nuclear Test Ban Treaty during the height of the cold war.

The wars all over the world: the war in Ukraine is on your hands, the destruction in Iran is hard to believe, you are not just evil, you are insane.

To be fair, I found that part consistent. There is limited cooperation between enemy states all the time, e.g. see the grain deal between Ukraine and Russia before it collapsed or the "red phones" between the US and the Soviet Union during the cold war.

In the case with China, the "cooperation" is much more extensive still, due to all the economic ties that both countries are currently unable to sever - which I think is also a reason that China is still seen as a "competitor" and not a full-blown "enemy state" in the US.

It's restricted to areas where there is a genuine common interest of course. In this situation, I guess the "other actors" would be terror groups, criminals or just reckless corporations - that aren't aligned with either state.

Obviously, such a cooperation wouldn't keep China or the US from developing models with those capabilities for their own armies.

--

There are lots of other takes with questionable logic in the essay though, such as that China is unable to train frontier models by themselves due to lack of hardware - unless they obtain the training data directly from American frontier models via distillation.

Or the assumption that open weights models will be completely opaque and immutable after their release, so a model that passed all the "safety" tests can never be turned back into an "unsafe" model. This seems pretty ridiculous when people are already finetuning open-weight models every day to add new abilities or remove restrictions.

And of course that China must not have those abilities because it's an Authoritarian Regime, but Trump USA is totally fine...

E.g. they might say something like "We absolutely agree that international regulation of AI is needed and we're willing to co-sponser an initiative with the US. One of the most pressing matters we see is the spread of despicable misinformation through unregulated models regarding certain events in China's history, the treatment of certain minorities or the status of certain provinces..."

But why call them overpriced? Compared to what? Even if we take the margin reports at face value, we don’t know their training costs, etc.

Curious if this was more of an emotional take or if there’s actual evidence behind it.

I guess the question is more he doesn’t want people catching up by doing cheaper training (through distillation or otherwise), and he definitely doesn’t want companies to spend their money training these models and then _giving them away_, which fundamentally undercuts their commercial model and any way to claw back their investment

Compared to some available Chinese model I guess. For most common tasks the additional intelligence is marginal and the cost is around an order of magnitude higher.

A much simpler summary:

- open models good.

- smart models _can_ be bad

- smart open models that can do biotech work are dangerous. worth the hassle of certification _if_ we can get everybody on board with minimalist certification.

- banning open models just in US is neither good or bad: is stupid.

Alphafold and others solving the protein folding problem are revolutionary.

An LLM can max provide an instructive tutorial protocol for lab work, but designing novel proteins is not it's job.

I am interested to enter the bioinformatics space eventually and the regulation happens at the DNA printer level. Anyone can design yeast nowadays that excretes heroin, but the DNA needed for the genetic modification won't be printed by anyone. As long as DNA printing tech is extremely regulated it's all fine, if home printing becomes a thing then on the other hand maybe people won't even need AI to print deadly pathogens at all. A database lookup will do.

"Dangerous to our bottom line"

"We call it dangerous to hype up its abilities"

And you know this how?