kalkin

It seems like the rest of the world should maybe not go on treating software as it always has: https://www.cnbc.com/amp/2026/10/23/php-vulnerabilities-ai-c...

And it also seems to me that prosecuting the people who warned about this 6 months ago for unspecified "flagrant acts" will probably not fix the problem.

A few people putting misinformation on the Internet - which isn't restricted to AI crawlers in any way, even the new disclaimer is something an AI model is likely going to notice more easily than a human since they read faster - is not going to stop AI capabilities from advancing.

We're in "something must be done; this is something; therefore it must be done" territory here except usually the something in question is merely useless and not directly harmful to bystanders in the way this proposal is.

It may happen anyway but you can say the exact same thing for AI development. In neither case does it seem like a good excuse for bad behavior.

https://talyarkoni.org/blog/2018/10/02/no-its-not-the-incent...

Just to be clear, you're proposing that we poison the internet with misinformation - which is just as misleading to humans using traditional information retrieval tools as to AIs - until it's completely unusable, and this will be good because it's bad for AI, and everyone who is misled in the meantime is acceptable collateral damage?

Here I'm quoting from the website:

> It should go without saying that you should never publish fake content. While my experiment relied on tens of thousands of unverified stats, I ran it in order to share the results with the public, and the website, while still live, now carries a disclaimer.

"now carries a disclaimer" sure suggests it didn't, so "you should never publish fake content" really doesn't seem like a standard the author actually held themself to.

I'm not making an argument for a single centrally-managed trust store. The thing that seems most obviously bad to me here, as a non-crypto-expert, is that any CA can issue a certificate for any domain. This is probably made worse because there are hundreds of CAs, and AFAICT functionally every trust store trusts all of the same ones.

I don't know how we fix this, but it does seem like a big problem.

This isn't, like, one misbehaving client. It's the way the ecosystem has evolved. GP didn't specify whether they were criticizing the standard or the ecosystem more broadly, but even if they were criticizing the standard, it's pretty fair to evaluate a standard at least partly based on how it gets implemented in practice. There's a reason cryptography tools for the last decade or more have been focused on being misuse-resistant.

Neat, never heard of those. I'm curious how this works with, like, acks, and also whether there's any public prior art for anyone having trained an LLM beyond one.

I'm curious how you would airgap a datacenter-scale training run. No reading training data from a cloud store, someone from OpenAI flies out to Abilene or whereever with a suitcase full of SSDs? When researchers in SF write code for a new RL environment, they ping somebody who sticks it on a flash drive and walks over to the server rack with the GPUs? Has anyone ever trained an LLM like this?

Maybe you don't mean a literal airgap, you just mean configuring Internet access off. But... they did that. That's why the agents had to hack Artifactory to get it...

Recall that multiple of the Chinese LLM providers got caught sending at least some of their queries to Claude. That would be a pretty funny way to get caught in turn doing something by the US government.

Why's that?

Is social media much better in 2026 now that there's been such a backlash against moderation?

The article says that scores began drifting down around 2012. Gemini is telling me that NAEP peaked in 2013. Elsewhere in this thread there's a claim of an inflection point around 2018 in PISA, which eyeballing Gemini's numbers lines up with NAEP.

Does this correlate with the racial proportion data?

"roma aeterna" is also the sort of alias that's associated with a particular set of political views. (Not conclusive! But a rational thing for someone to notice.)

Honestly I would find a certain satisfaction in a world where the next Democratic administration takes away Fox broadcast licenses, forbids use of Oracle by any government contractor unless Ellison hands CBS over to Rachel Maddow, and tells Bezos that Amazon is going to be banned from public roads unless the Washington Post front page has positive stories every day. But I doubt the same is true for the many pro-Trump people who complained vociferously about Biden officials expressing preferences to social media companies because of the merely implicit threat that they might take government action of some kind if they weren't listened to.

So your argument is that the government is punishing Anthropic for its public statements? And it is correct to do so?

(I think it's also the case that Anthropic didn't make any public statements about the negotiations until after the DoD was already publicly threatening this designation. But that seems less important than the question of: is it good for the US government to try to destroy companies whose public statements it dislikes?)

That's true legally, but the government's public statements did initially very explicitly present this as a ban on any dealing.

"Anthropic a Supply-Chain Risk to National Security. Effective immediately, no contractor, supplier, or partner that does business with the United States military may conduct any commercial activity with Anthropic."

Presumably, according to the defenders of the designation in this thread, this was an innocent mistake that doesn't indicate any political intent.

Nobody is suing to force the DoD to buy anything, so maybe you shouldn't be accusing others of failing to refer to the topic of discussion.

The DoD is free not to sign contracts. Having them be able to get a company blacklisted from a whole sector just because they couldn't agree on terms with that company is obviously very different than that.

The idea that "fair for the military to object to" implies "fair for the military to ban from their supply chain in any capacity no matter how unrelated the use" is so outlandish that it's never been done before (to a US company), and I think we can be confident that the DoD has frequently objected to would-be contractors' terms.

The claim that it being done for the first time - by a Secretary of Defense denounced the politics of the company in the same text where he announced the designation - isn't "political", is something I have a hard time believing a functioning adult who isn't employed defending the government in court could adopt.

But hey, maybe in some language this is a "textbook" official statement with no "politics":

  Cloaked in the sanctimonious rhetoric of “effective altruism,” they have attempted to strong-arm the United States military into submission - a cowardly act of corporate virtue-signaling that places Silicon Valley ideology above American lives.

Why has this never previously been done to an American company from which the DoD decided not to purchase something?

Perhaps people who are noticing politics here - including in the decision of two Trump-appointed judges, against the he vote of another Republican-appointed judge- are not the ones who are choosing not to see reality?

Why has this never previously been done to an American company from which the DoD decided not to purchase something?

What is the evidence that Anthropic said this?

As of writing it still says:

> For Anthropic, Google, and Meta, the catastrophic breakouts happened inside the testing environments of the exact same contractor.

If this is the level of understanding you have of the relevant incidents, there's a lot of chutzpah in saying that other people are "selling garbage", carrying out an "extraordinary confidence trick", etc.

The people who made the LLM revolution happen were by no means exclusively rationalists--but more importantly, lots of people have lots of dreams about the future and even when pursued very, very few of those dreams pan out in anything like this way. LLMs are not what they are because of rationalists' sheer force of will.

Also, nobody ever bothers to argue why a specific proposed regulation is "regulatory capture" or would burden startups more than big companies or anything. It's just supposed to be obvious that corporations love regulation and it's bad for the public, all of post-WWII political history notwithstanding.

I don't understand why strict liability is supposed to advantage smaller players, unless the law is like, specifically Anthropic and OpenAI are responsible for what people do with their models but other model providers aren't.

- If this applies to people who release open weights models, that becomes a terrible idea as long as you're subject to US laws.

- If it just applies to people who host them, that still probably advantages bigger players who can afford in-house legal and won't be destroyed by losing one lawsuit. Or maybe we create some kind of AI-misuse insurance analogous to malpractice insurance, that smaller players can buy in to? But that takes time even if the finances work out at all. And, uh, I'm not sure malpractice is a model we should aspire to in other industries.

Plus, presumably an immediate impact of this is that hosting providers all have much stricter safeguards classifiers. And the fact that somebody else is deciding what you're allowed to do with the model is one of the things that seems to make HN angriest at the frontier labs in the first place...

To be clear, I think this might be a good idea! I think all of the possible downsides I've listed are pretty small potatoes relative to what happens with no regulation of AI at all. But I'm pretty sure that if the big companies were proposing it, people would be calling it "regulatory capture" too.

Again, I think debating export controls is reasonable. I'm not sure I'd trust the CCP with control over powerful AI any less than the Trump administration.

But that's different than saying we don't need new regulations at all. Even strict liability of the form you seem to advocate would seem to need new laws. And it's not clear to me how it's consistent with your aim of just regulating what people _do_ with intelligence. Presumably AI companies would clamp down much harder with restrictive classifiers in that world--including hosting providers for open source models, if they also assumed liability. And then the people with unrestricted access to this technology are only those who can afford home GPUs, which are going to be intrinsically less efficient because they can't batch queries...

I think there are genuinely novel things about AI vs human intelligence which pose genuinely novel problems that would need genuinely novel policy to solve them.

Ah yeah. I think the economic value of AI, unlike bioweapons, is probably too high for anything like that to be viable, but I think it's a reasonable thing to aim for. (Dario probably doesn't because he's a believer in short term positive biomedical impacts of AI in a way that I'm doubtful about.)

I can't even tell whether the poster to whom you're replying is saying that "let's uh, stop" is profoundly naive, or that failing to say it is profoundly naive... I've certainly seen both takes elsewhere.

Hmm, I clicked that link and the first claimed bad prediction I see, from 1996, is "singularity 2035 (actually 2025)".

Predicting 2025-2035 as, at least, the period when AI becomes a really big deal, seems pretty good, even if the jury's still out on "singularity".

Broadly, the rationalists seem to have been pretty early to realizing LLMs were a big deal, and certainly seem to have had a much more accurate picture of how they'd develop than the people who denounce "TESCREAL" and talk about "stochastic parrots". It's fair (and IMHO correct) to ding rationalists for lots of things, but specifically poor prediction about AI seems like a bad one, insofar as anything has been tested so far.

Do you think Anthropic's behavior in the last year is well explained by aiming to be "the favorite child of govt"?