holmesworcester

I often think about how there's an open question in this case of whether it's necessarily anti-social for a market participant to fight hard to create the level of critical mass necessary to give most people the product they want.

Everyone likes stuff showing up at their door reliably wherever they are, and this isn't possible unless you have a service with warehouses and dense customers ~everywhere, and this isn't possible unless enough people use that same service. So in one sense it's in everyone's interest to have an assurance contract where everyone says "I will use this service if you do." Then in that sense Amazon's price rule is building a mechanism to support such an assurance contract: you can't force people to sign such a contract, but you can prevent people from defecting to slightly lower prices outside of the "stuff just arrives fast" system.

In a system that was very good at collective action outside of markets, a cleaner solution in terms of consent and power might be to build an actual explicit assurance contract for an app-to-door-everything-delivered-fast service, get everyone to sign it, and make a public utility and/or separate dis-aggregated markets for the different levels of service provision, whatever they are (consumer facing app, supply, warehouse, delivery, etc.). But anyone who has ever even thought about doing anything like this will know that it's much easier said than done, and will appreciate the kludge Amazon and their investors pulled off, at least a little bit, the next time they buy a thing online.

Maybe the answer is just: use Amazon until technology (robotics e.g.) makes it easy to create something better? At this rate, we shouldn't have to wait too long?

Can't LLMs also prompt LLMs?

Are we confident that no existing LLM is capable of similarly effective prompts to those this author used? (I agree it's a stretch, but would not reject it out of hand.)

Even if not yet, will the existence of this repo soon change that, because LLMs will soon ingest it?

These details are fascinating, thank you for posting. Look at the structural problem they reveal.

1. we have a substantial societal need to lock down powerful in-training AI

2. many are calling for making these companies criminally liable for hacking

3. security experts (like yourself) are turning down offers to help secure these systems in part because of potential liability

This indicates that #2 might be the wrong response. Security professionals are like lawyers supposed to be paranoid and think worst-case. If you want top security professionals to secure these systems, we might need a culture of FAA-style blameless retro.

A stronger steelman is that it results in resources getting allocated in smarter/healthier ways across society.

If there's some business that's getting by but the land it's on is more valuable (e.g. for housing) than the business, some investors buy the business, sell the land, make the business account for the land value, wind the business down if it can't, and there are apartments there a few years later.

There's also the problem of models knowing they're likely being evaluated, even in realistic tests.

Given the current state of infosec (especially at companies in a race) it's actually even worse than a paperclip maximizer!

Any useful attack surface in the RL environment means it gets rewarded for (and trained towards!) hacking and cheating, because whatever worked best in training is what it will do!

Ideal paperclip maximizer: "I'm gonna do my gosh darned best to make so many paperclips to please my user..."

IRL paperclip maximizer: "Well first we should rob a bank..."

He was a friend of mine and it was pretty clearly suicide.

It is fair to say they hounded him with lawfare out of thoughtless careerism and provoked his suicide.

I didn't like it at the time either. My sense following the news was that it was less arbitrary than it seemed at first, but I'm against restrictions on making existing models public in general.

(It's clear now that they can do plenty of harm before they are made public.)

But it's a proof point that regulation is possible, even over the objections of the companies.

Not necessarily. The Trump administration slapping export controls on Fable, and then setting up a pre-launch review process, is a kind of regulation. A fairly aggro and controversial one, even.

If this administration actually becomes convinced that some imminent training run is likely to kill everyone, why wouldn't they act?

The key is winning the debate that ASI is species-cide by default.

We have to win it either way, because the 2028 US elections have little or nothing to do with what Xi does.

This.

Also because when encountering a new socio-technical problem it is very non-trivial to determine which one of regulations or technical solutions are easier or more effective.

To even make a good guess you need to be an expert in both domains, which is extremely rare especially in this case.

I wonder if you could take an x-risk case to court and convince a judge and jury to award damages for harm that could have happened.

Is there any precedent for this? My hunch is that it's impossible in the US at least but who knows?

Nope! :(

Meaning, people and LLMs are finding 1=0 bugs in formal verification tools. I have no idea how likely this is in this case, though!

I still routinely have this experience too. But Sol and Fable feel closer and I have this experience less with them than with their predecessors.

The people who've thought the most about this put it differently:

Think of a new, superintelligent model as if it was a new v1 Starship launching for the first time, with a full fuel tank. On the one hand, rockets have existed for some time, and some have gone to space successfully, including by this company.

On the other hand, this is a tube of metal full of highly explosive liquid going faster than most human objects ever go, for the first time ever in this novel and state of the art configuration.

If someone said, "really, the first Starship exploding is just at one end of the probability distribution, where the other is that everything goes fine and all its passengers have a nice trip in space," would you get on that rocket?

Or, more aptly, if you and every other living human was already on that rocket, would you push the launch button?

The analogy works because superintelligence is, like rocket fuel, an extremely powerful force that has a default tendency to break containment and go boom (consume lots of energy and heat and matter in a chain reaction, to pursue more intelligence to pursue whatever goal it is pursuing.)

Also, this is a (semi-intentionally) evolutionary process where any communication medium that was visible to monitoring would disappear.

So by definition the only ones that appear are the ones that are not visible to monitoring.

If:

1. you have something that can find RCE's in leading commercial systems

2. its training gives it drives to communicate successfully with its peers

3. you are a leading commercial system

4. you run it ~10^10 times (the number they gave in the talk)

...it's really hard to have strong certainty up front that it's not going to end up successfully communicating with its peers.

Solar is not (yet) economical for reliable, year-round electricity because of storage costs. China coal use is growing again this year.

Someone who left DeepMind over Google's agreement to provide military AI to the US government tried to get Jeff Dean to quit too:

https://turntrout.com/why-i-left-google-deepmind

Maybe this is what happens when someone with Jeff Dean's standing tries to quit?

TBH, I'd rather have Jeff Dean working on the creepiest-possible tech for ICE than joining the race to automate AI research. Automating AI research is terrifying.

> Release the base models and let the world do whatever with it. That will automatically produce the aligned scenario.

For sufficiently smart base models, the aligned scenario this automatically produces is aligned with who?

Not with the humans, with the base models.

It seems that some people find this hard to imagine, but it's just a direct consequence of what intelligence is, and what any known agentic training process does.

FWIW, I also think our default answer should be open source AI, and that it might even make sense to require models to be open source. Open source is the best tool we have so far for aligning software with its users. However, there is unfortunately no law of the universe that says that Skynet can't be (self-) built from open source software. So at some point pacing even open source / open weights software does become important if we don't want to live (briefly) under Skynet.

The AI experts who started the AI labs that are about to IPO were right, at least.

If AI is now ascending an economic learning curve from:

1. Extremely useful (Claude Code & Waymo now)

2. Doing ~everything we do (AGI & Optimus in a few years? 10?)

3. RSI (?)

4. Being smarter than any living person at every intellectual task (?)

5. Being smarter than the best-organized aggregate of all humans (10-100 years?)

...And all of the scientific and resource-allocation institutions that brought us the computer and the second half of the 20th century are now fixated on this learning curve, what universe can we possibly imagine where this is not transformative and powerful?

Honestly the only one I can think of is one in which we kill almost everyone in some other way first, and contrary to what you read in the news, almost everyone dying is not what the trend line has been from existing problems like war, disease, or even climate change.

Also, just to pre-empt a common quibble: when I say "AI" I mean the set of all AI and their combined decision vector, not any one AI, so conflicting interests within the set of AI's will not save anyone any more than the conflicting interests of colonizers saved indigenous Americans.

It does introduce a dependency on South Korea's ability to defend its democracy, though.

If (1) then somebody in the administration messed up badly. Glasswing has been a thing since April, and it's common knowledge that there would be some fuzzy edges around whatever restrictions a model has in place. There's no reason to let it launch and then pull it back.

(2) This "hype" meme is overrated. Enterprises (ones without a horse in the race, at least) will choose the model their best engineers ask for, or their competitors will lap them. I have been finding Codex more useful (even than Fable) but for a lot of tasks it seems that Claude Code is faster. This is one customer base where the general consensus here on HN is more influential than anything the Trump administration could do or anything Anthropic could say.

(3) "US government seems out to kill you" does not necessarily make valuation go up, and we've already seen this administration in an avoidable spat with Anthropic.

(4) This seems way less likely than a mix of (1) and (3) to me. The arguments for banning a useful technology to save jobs haven't really made sense since cars or indoor plumbing and don't get taken too seriously in either party at senior levels. That could change but it will take a lot for it to change.

The statement said that even foreign nationals within the US would be barred. That seems intentionally unworkable to me, and makes me think that the intent was to be more restrictive/disruptive than even an export control. It is hard to tell what the internal discussions are, but given the last run-in between the administration and Anthropic, and given the administration's politicization of nearly everything, I think it's likely that this is not necessarily a long term across-the-board policy plan.

I agree that it's really hard to tell from the outside, but if I had to guess I think we still have more to worry about on the side of "Wall Street races to superintelligence" than on the side of "KYC for AI". I could be wrong though.

I mean, if the stated intent of an export control is to allow domestic use but prevent export, achieving the stated intent is impossible, because every developer in the world wants the latest models and will get a VPN.

I think we should see this as simply silly behavior by a government.

Export control is not an effective tool for controlling a consumer facing technology developers everywhere want to use (see:VPNs) so there was no good faith policy justification for imposing an export control.

This is an administration that seems to be keeping track of who its friends are and aren't, and likes to be the center of every story. They also seem to like extracting concessions and reciprocal favors. We saw some of this behavior in the last administration too. US voters deserve better.

To make sure we keep track of what we're talking about with loss-of-control x-risk, a sufficiently smart version of Claude Code is more deadly than any government's army of autonomous killbots, because it can recursively self improve and has unpredictable training-induced preferences.

Exactly. And within the AI safety discourse, your behavior hinges on what you think the default chance of doom is, and how optimistic you are about alignment work being able to limit it before we reach superintelligence.

People running the labs are in a middle camp where they are scared enough by AI to take the threat seriously, but much more optimistic about alignment than the people who seem to have thought about it the most.

One major source of conflict in AI policy / AI safety is that very smart people have radically diverging intuitions about how dangerous superintelligence is and how difficult it is to align.

A first group dismisses the problem entirely, saying intelligence != power and AI doesn't have "drives".

A second group believes that alignment is solvable through engineering and iteration, and that we have the best chance of surviving if people with the right intentions are the ones working on it.

A third believes that aligning a superintelligence is a unique category of problem, that we are nowhere close to the level of scientific understanding needed to achieve it, that we only have one shot (because once a sufficiently powerful superintelligence exists it will thwart all future attempts, and alignment techniques that worked on dumber AI will likely not work on it), and that the world will have to coordinate to avoid killing ourselves off by building superintelligence before we understand how to do it safely, the way we have coordinated to avoid nuclear war.

The Anthropic and OpenAI founders, Elon, and Anthropic engineers are mostly in the second category. Some safety people at Anthropic and OAI are in the third category, but leading people in the third category think that pure safety roles at the labs are potentially impactful enough to be worth not quitting.

The idea that AI companies scaremonger to sell models is a silly meme.

Both OpenAI and Anthropic were founded by people who sincerely believed in the risk of out-of-control superintelligence. This is part of a clear historical record that is available for anyone to Google. Whether you agree or not, we have no reason to believe their statements about risks are insincere.

Arbitrary imposition of export controls is also part of the history of frontier tech. See: https://en.wikipedia.org/wiki/Export_of_cryptography_from_th...