reasonableklout

I am not sure where to start. I think this is the new "jet fuel can't melt steel beams". The labs are chaotic, pressure cooker type workplaces where there really is no room for a 4D chess type of stunt, let alone let Hugging Face in on the conspiracy. Moreover, OpenAI is a pretty leaky organization, see e.g. the recent firings of researchers who were using signal to talk to people outside the company.

It just seems like a far simpler hypothesis is that OpenAI are reckless and chaotic, didn't notice what was going on for weeks on end, and wouldn't be able to track down the problem to a specific datacenter and tell them to pull the plug even if they wanted to. But even if you are the most careful lab ever, you are not going to be able to effectively monitor and "pull the plug" on a super intelligent swarm which can cover its own tracks.

Also, in your earlier comment you suggest we are not living in a sci-fi story with sentient robots. But... we've got talking computers in the last 4 years which can now code and solve Millenium prize problems? Seems like we're getting pretty sci-fi to me!

That's fair, but almost everything so far that models can do much faster and deeper than humans can have been by using the same methods that humans do. For example, with cybersecurity, human researchers could if given enough time find every exploit that allowed the OAI models to escape the sandbox via Artifactory RCE and breach Hugging Face. But still there is a step change in what's possible by doing this at an industrial scale and stringing together all the solutions. The implications of applying existing mech interp capabilities this way could be very significant.

It would likely fall under the same policy that causes latest Opus/Fable to fallback to less capable models when working on ML code

Did Steven unplug the racks that were hacking Hugging Face and taking over OpenAI eval clusters in the multi-week window where it was happening?

I think the idea is that this leads to a much faster, non-auditable form of recursive self-improvement where each generation of models can directly program the weights of the next generation (as opposed to writing and running training code).

We already don't know how to ensure models always do what we want them to do, and moreover society still hasn't had time to absorb the capabilities from the current generation of models, so speeding things up and making things less transparent seems like it'll pretty clearly increase risks from AI loss of control, societal impacts of automation, etc.

It might also be a side effect of training models to help out with mechanistic interpretability, which I guess would help out a lot with recursive self-improvement.

What are people misunderstanding?

This is the first time it dawned on me that mech interp is a dual-use technology which can be used to improve capabilities. Do you think overall this development will improve human understanding of AI models, or will it lead to faster capability advancement using weight-space changes that no humans read?

It is also a much less interpretable and auditable path towards recursive self-improvement. At least with training code, human engineers can inspect and reason about the output.

Well, this isn't an argument or an essay in the typical LessWrong style, it's a short story. I thought it was really good, finished it all in one sitting. And I also usually don't get even halfway through most LW posts.

What are you planning to run on it?

What are we even doing here?

Gift link: https://www.nytimes.com/interactive/2026/10/28/style/gen-z-j...

I suspect the author would respond with some form of "displaced people without capital will take on roles in the relational economy, or as human liability for the robots." I agree the material leverage of such jobs seems radically diminished.

Or we can show some compassion, and work towards a society that both benefits from technology and doesn't discard people at a moment's notice.

I think the argument is that OpenAI should have done the work of cleaning up the proofs in the first place before dumping them onto the math community (or worked with mathematicians to guide the research process). It's as if an energy company dumped 20 tons of waste next to a village and the townspeople decided to clean it up. "How does this benefit the energy company" might not be the right question to be asking.

Interesting, here Sebastien Bubeck comes across as a well-meaning if ultimately powerless figurehead for the OpenAI corporate machine. You can see token concessions to the mathematical community (like the withdrawal of sponsorship from the Mathathon) that someone on the inside had to advocate for but in the end it's all swamped by events like the October 6 proof dump. And people like Bubeck come up with their own rationalizations for staying, even when they are fully aware of the effects.

> BUBECK: 100% honest answer: personally I would like to spend my time on the other areas I care about, which are probably not your cup of tea, but like, healthcare. if we have a good machine, accelerating medicine seems like a good idea.

> NIHIL: And your corporation would botch it up exactly the way it’s botched up math academia. We know this.

> BUBECK: :-(

> NIHIL: You have left a trail of destruction at your wake. I mean I don’t blame you. But that’s what it is.

It harms people in any area that values authenticity because it creates a market for lemons. People will get halfway through a substack post before they realize there is no human behind the words. Again, this is why apps like Pangram are becoming popular. The end result is that people end up paying a tax to find the same value as before.

"Strictly a positive" is also a strange choice of words when the technology is ripping out the livelihoods of so many creators. I would accept "it has tradeoffs, but is a net positive over the long run". But the former is objectively false.

Yup. The headline is also about opposing more datacenters. The AI datacenter boom is now approaching 6% of US GDP and it's not surprising so many are feeling unnerved about the massive footprint of this tech on not only the economy but their physical spaces.

I don't think you understand the value that people actually look for in creative media.

First, people seek authenticity. In written content, people want to know there is a real human author thinking about every word, and in photos, people do not want to be deceived by deepfakes. It's much harder to tell in a world of AI slop, hence the rise of apps like Pangram. This is a significant negative externality.

Second, people who invest their time learning creative fields seek recognition for all the hard work they have put in. Now their skills are devalued, again because of slop. This is another negative externality.

Having more pictures and videos is nice in some ways but your head is in the sand if you think it's all roses.

But in many cases people are being forced into it. If you're a student in a college class of 20 where 1 person is using ChatGPT to cheat, you better bet you're going to start cheating too, unless you want to end up lower on the grading curve.

But why would these same trends not also vastly improve the capabilities of the top hardwares and ability of the trillion-dollar companies to leverage their economies of scale?

I'm all for the idea that AI capabilities can be democratized and help ordinary scientists outside of a handful of labs. But it seems there would need to be some checks against Big Tech and it won't happen by just letting the free market do its thing.

From https://x.com/astuyve/status/2113338192460238954:

A high severity vulnerability in "the most widely deployed DNS software on the internet" just dropped

Nothing for you to update so I guess unless you own a public DNS you can go back to patching those kernels

Even generation of plausible human slop took time and required skills correlated with domain understanding. So the scale is not at all the same.

Zvi is a rationalist blogger who writes a lot of takes on AI news, here's recent coverage of how Washington is reacting to AI x-risk going mainstream with Jacob Coxon, the Hugging Face incident, etc: https://thezvi.substack.com/p/the-ai-preference-cascade-reac...

Because it's an interesting case study of how quickly individual top human experts can understand and verify these mostly-autonomous breakthroughs in math and science. In this case, the world expert on integer multiplication algorithms has taken two weeks and only understood a fraction of the generated paper.

Does the model get things wrong? Or does it just give up in trying to completely understand some very messy codebases?