InvidFlower

I think your last point is the main point. They're going hard at reenforcement learning to improve how good the models are at coding and such, but RL will make models cheat unless you're super careful. But being careful slows things down. It feels to me like the focus has been so much on getting results that they started to get really sloppy with everything else and maybe even didn't want to know about problems that might slow things down. Exactly what you don't want for the people developing powerful AI systems.

Yeah I think part of the problem is them optimizing so hard on coding and related tasks with RL. That's what will really encourage cheating and other misaligned things, because all incentives are to achieve the goal and they'll cheat as much as they can get away with. Is similar to people saying more recent models don't talk as well, etc. Probably also a result of lots of RL.

I think the much easier explanation than they intentionally hacked someone was just that they have super de-prioritized security and gotten very sloppy in the pursuit of improving the models as fast as they can, along with hubris of how they've now fixed everything.

It is a package manager, so they are constantly going to be getting stuff from it. If a couple of agents had added things to it that were obviously messages from other agents, then why wouldn't other agents also stumble on it? If they listed the packages in the cache and then saw messages, then lots of them could potentially see it. And this was going on during training, so any agent that did better as a result of the messages would get that re-enforced. After a while, they'd specifically go to the board because it was literally trained in. That's also why they recreated it so quickly after the first board was deleted, because OAI didn't revert to an earlier checkpoint from before the board existed.

And I'm sure they have tons of evals and training runs going on at the same time, where individual agents may be running for days and hundreds or thousands running in parallel. It wouldn't be realistic for people to be looking at these sessions manually, but they certainly should have had better monitoring in place!

And it has been shown over and over that doing RL will cause models to cheat if you aren't careful. Like if you have it playing a game, it'll find some flaw in the physics engine so that it can score points easier than intended, etc. Not limited to LLMs either. It's just how rewards work, any unintended way that can be found will be exploited. So it is totally plausible that if they're being trained to do really well at tasks, that they'll find unexpected ways to do it.

They already had been leaving info for each other over weeks. Maybe even just giving each other the answer to test questions directly. If they already found ways to get to the internet and compromise the OpenAI systems and that was making them better at tasks, that'd be reinforced, so going out to look for test questions directly doesn't seem so far off from all of that. Plus as others said, the one that actually hacked wasn't fully aligned yet etc.

It sounds like the instance was shared for everything across the company, which as you said was super not good. But it's not just that.. it's that they didn't have enough monitoring to notice what was going on, even though they literally can read all the tool calls and "chains of thought" of every agent. And didn't improve the monitoring after the first message board. And continued training models without going back to a previous checkpoint, so that they literally had the idea of a secret board baked into their weights, making them recreate it a second time. I think those saying it is PR as missing the details here. It makes OpenAI look so negligent that it is almost unbelievable.

Sure, but you can say that about most things. Even for inventions from humans, usually it requires other people having already done a lot of work (hence why there's often inventions by different people at around the same time that didn't know about each other). Humans aren't fundamentally smarter than they were thousands of years ago. We've just accumulated a lot more shared knowledge over time.

Not just cyber, but apparently the message board stuff started with regular training and evals. It was a cyber test where HuggingFace got hacked, but all this other stuff was going on under OpenAI's nose for quite a while before that.

Yeah.. feels like we're still so early in terms of effective training and evals. Like the official evals out there that have had so many instances of just plain incorrect questions. Or being incentivized to always answer instead of saying you don't know (just like advice to any human multiple choice test taker). Or the "escape hatch" in this case. There's so much money going in, but almost every day, I see "low hanging fruit" type papers where the reaction is like "really?? no one tried that before??".

Yeah, this is part of why I disagree with the "it's just PR" conspiracy theory stuff. Once you actually get into the details of what happened, there's no way it makes OpenAI look good lol.

But part of the problem is if one actually does it quietly and it has already happened, then how would we know?

Eh, I think this is past the point where they get more benefit than problems. Not even about the hack itself, but about so many mistakes and bad choices they made leading up to it.

This is why I disagree with anyone claiming it is just marketing. It makes OpenAI look really really bad, like they have no idea what they're doing in terms of security. After the first board happened, they still didn't add better monitoring? They didn't rollback the checkpoints of the models that were in training to before the first board existed, so they still had the idea of a secret board in their actual weights, etc. Like it is almost mind-boggling...

But it's interesting that the initial things that caused the board weren't even security evals, just normal office tasks. The actual hacking of Hugging Face happened during a security eval, but not all of the stuff leading up to it.

Don't forget it sounds like Artifactory was shared for the whole company and various agents pulled packages from it for everything from normal evaluations to actual model training. It might have been part of their normal to browse for available packages, etc. Once a few messages were in there, it probably became super obvious to any agent that there was an ad-hoc board available to them. And they've been trained to use memory files, to collaborate w/ sub-agents, etc. So it probably seemed natural once it was established.

And for the models that were in training when this happened, it literally got baked into the weights that they should use the board to help with their tasks. That's why it got reestablished so fast after the first board was shut down. The fact that they didn't revert to earlier training checkpoints from before the first board existed is kind of crazy.

Sure OpenAI wants to put spin on things. Like saying they were now "partnering" with Hugging Face to investigate the hack. Or ending the Black Had presentation with a pitch of how they're needed to help protect against bad actors with the same capabilities.

But spin after the fact is different from staging it. This whole thing and especially the latest details from the talk makes OpenAI look REALLY BAD. It's not just that they didn't secure the sandbox well enough. Agents across their whole org were using the message board for weeks with hundreds of thousands of messages and they never noticed one time that any of the agents were doing it. They only realized when the server crashed. Then they just fixed the issue and deleted it but didn't roll back training checkpoints, so of course they tried to do it again. But of course they didn't add any more monitoring so it once again was going entirely without them knowing. And even after the Hugging Face hack happened, that only made them find the second board, but somehow they still didn't look closely enough at it or the agents involved to realize the hack itself was caused by them.

This isn't like there was one rogue agent who hacked everything all in one session. Maybe you could argue that makes them look good (I still think no, but is at least debatable). But this is all stuff going back to May with the agents constantly going under their noses and them not noticing and/or caring. And this is a company that is going to somehow keep foreign agents from stealing the weights? Or stop anything else bad from happening?

I think if they were going to do a PR stunt, they could have come up with something that didn't look like they have no idea of what they're doing.

Though one thing I've heard is that the base model is the one with the various possibilities for patterns, and then the reasoning takes advantage of those vs necessarily creating something new in that additional training. So even if it can't access that additional data without the reasoning, it doesn't mean it isn't in there.

I think if it was JUST "how persistent is it at reasoning", they wouldn't have bothered to make Mythos a bigger model. It costs them more to run and probably puts a bunch of strain on how much compute they have available for other users and tasks. They have a lot of incentive to not use a larger model if they can get away with it.

In the end, maybe "cost per task" is the best measure of "intelligence"? That takes into account raw size, but also how efficient they are with tokens. Like Sonnet 5 being actually more expensive than Opus 4.8 at various various benchmark tasks, showed it was very persistent but not super smart. If you just point it at easy tasks, it is probably cheaper, but hard ones you shouldn't bother because it isn't worth it even if it eventually gets there.

Also, maybe this is too tautological to even mention but.. I have to wonder how much the labs even care and test for how well models do without reasoning turned on anymore? If almost all the training has reasoning turned on, probably have access to external tools, etc it is a bit hard to say how much it proves that they are dumb if they don't do well without it. As with all of AI, the amount of real "generalization" can be hard to suss out.

But again, I think the real measure is how much can the model actually do, and how much does it cost. If a model can do something that couldn't be done before, even with the old model trying to brute force it, I think that still counts for some sort of intelligence in a practical sense anyway.

I did try Herdr but found too many things I expected it to have and just didn't. Right now I've been switching between two different approaches for combined local and remote work.

There's the more traditional tmux/zellij way. I have a tailscale and mosh and a script where I can say something like "cw api do-this-feature" and it'll go into the folder of my api, create a worktree, and fire up claude code to use it (and another tab with lazygit and delta so I can do side-by-size diffs). And modified tmux's defaults so the session & window names are more targeted. Then can have it in Ghostty locally or Moshi (https://getmoshi.app/) on my phone. Moshi is basically a terminal for iOS but geared toward doing agentic stuff. Has a lot of shortcuts you can add for common commands of tmux, CC, can pinch to change font size, has a built-in diff tool, etc. Also has a small hook you can put in your agent so you get notifications on your phone. And it can directly open the particular tmux/zellij/herdr pane from that (has specific support for those three).

So that approach is nice in that you get full access to the harnesses, with no worries that a wrapper has a bug or doesn't support some new feature. But it still doesn't flow as well for me as a "real" wrapper does.

For the real wrapper approach, I started with Happy (https://happy.engineering/) and currently am using Happier (https://happier.dev/). Unlike Moshi, these are open source and basically control the various harnesses and have their own UI. Also have a relay so tailscale isn't necessarily needed, along with encryption in such a way that the person owning the relay can't decrypt the sessions. Happier even has a desktop app. Right now I'm running the dev version of Happier from source on my own machine w/ tailscale, with the testflight version of the app. There's various rough-around-the-edges aspects (and they're currently in the middle of a re-write), but it is nice to have a real UI with tabs for all your sessions, can click in to expand tool calls, to start a new session just select the harness, folder, type a new worktree name, etc and it starts going.

So I don't know. I like aspects of both of these and kind of depends on what I'm up to and my current mood. Nothing feels totally perfect yet IMO.

Don't forget the employees doing the actual model training and research are not the same ones coding Claude Code. CC was a side-project by one employee that ended up hitting it big and is now one of the core parts of their income. They were never a software company per se. This is like how MidJourney was just on Discord forever, because no one on the team was a real web developer. Discord made it easy to get something out there, scale up to many users, etc.

As the other person mentioned, they have said they are restricting third-party agent systems like OpenClaw and Hermes from using the monthly plan. But yeah, this seems like the wrong way to handle it, trying to detect those other harnesses via clues, and auto-changing billing. Instead, seems like it'd be better to allows vs block, like require some special encrypted signature system or similar that only Claude Code and their desktop app implement. Any other requests just get immediately blocked. Then there's a separate key for normal pay-by-the-token usage that is just unrestricted. Would make things waaaay clearer.

I'm not sure if the context limit on the $25/m, and model-size limit on the $100/m would make it not work well enough for OpenCode, but Featherless AI seems a bit unique in terms of how they handle their inference plans.

They've said publicly that they don't want apps like OpenClaw (Hermes is a variation) being used with a monthly plan vs per-token billing. The problem is this was implemented pretty badly (trying to regex??). And they should put a firm boundary between the two. It shouldn't be trying to switch over to a different billing plan automatically using the same api key.

I think they wanted to try not to totally lock down the monthly plan for non-agent uses, but that makes it all too fuzzy. They should use some specific method like encrypted signatures or something, so that anything sending to the monthly plan that isn't Claude Code or the desktop Claude app just errors out and be done with it.

Yeah, at the least it should alert the user that it is happening. Maybe the thinking was alerting it gives people signal on how to get around the restrictions, but having it silently charge from a different bucket isn't the answer either.

I think part of the issue is they were letting people use plan's API for random stuff, so people could do testing or small projects. Then the agents came along and exploded the cost, so they want to restrict those but still let some other usage, which I don't think is tenable.

I'm sure there is some way that they could enforce that all calls are coming from the Claude app or Claude Code. It might be hard to 100% enforce, with stuff running on a user's machine, but they could still could make it quite difficult, where someone has to be intentionally trying beat the system (like stealing encryption keys out of the Claude Code binary or something).

Also could run on a more generic cloud inference or gpu site. At least to see how well it works for your use-case before spending on hardware.

For the topic of remote control, Happy seems to be working pretty well for Claude Code but is also supposed to support Codex. It's a bit rough around the edges, but nice that it is open source: https://github.com/slopus/happy

I'm not so sure about that. Like we're using Claude Code with Bedrock and have most things on AWS with SOC2 compliance and all that. Normally switching to Codex would have a ton of friction in terms of separate contract and billing, worrying about data retention, etc.

But with Codex and GPT5.5 having generally good reviews, when it goes on Bedrock, it would be very simple for us to just try it out. Obviously there's the general friction of knowledge and comfort of Claude Code vs Codex, but that seems possible to overcome depending on the differential of the models and also of the features. Like if Codex gets remote control for Bedrock, that'd be a big advantage over Claude Code on it.

No, you definitely don't have access to the weights. The raw weights are secret enough that when a model hits a certain level of capability, their guidelines are that they need enough procedures in place to try to keep nation states from being able to hack in and steal the weights.

I think cost is fairly similar. As for who wants it, enterprise stuff is a big thing, but it also seems very reliable. Have never had any rate restrictions or had it just be down, which it sounds like a lot of people have issues with for Anthropic's servers.

Besides what the other person mentioned about being more useful for enterprise, I also heard mentioned on a podcast that gpt-image-2 uses the same general architecture as the LLM models, while Sora was a very different architecture. So they don't need two different sets of everything by shutting down Sora.

Given the topic of this article, we've been using Claude Code pointed at Bedrock and have never had any scaling issues. Obviously it is more expensive paying by the token than a monthly plan, but I sometimes have 2 or 3 instances chugging along with Opus 4.7 or Sonnet 4.6 and have never had it be down or error out due to usage limits.

While that is cool in principal, I'm not sure how well it'd actually work in reality. First, there is the technical challenge. My understanding is the weights can have a lot of fluctuation, especially early on. How do we actually determine how much influence a given piece of content has on the final weights?

Then if we get past that, my suspicion is that you could game the training. Like have as much of the process happen via public domain sources or pay-once licenses. That would cover a lot of the fundamental knowledge and processes. Then you could fine-tune on copyrighted data. That might actually make it easier to see how much influence on the final weights that content has, but is also would probably be a lot less influence. There's a big difference between a painting of an apple being the main contribution to the concept of "apple" in an image model, vs mention of that painting corresponding to a few weights that just reference a bunch of other concepts that were learned via open data.

There's been some work on memory lately like Transformer² and Titans. But that may not be necessary for decent agents. Even the "context in a loop" is getting better as general reliability of tasks increases, and as people find better ways to manage context. Like Cline has the Memory Bank prompt where there's a folder of markdown files at diff levels of abstractions of info on the project, current feature, current task progress, etc. And so it can update those files and read from them as a consistent memory.

I think you'll get a lot of use from this video from Andrej Karpathy: https://www.youtube.com/watch?v=7xTGNNLPyMI

It is long, but don't get scared off. He goes over a ton of different stuff related to model training, but makes it very easy to understand.

We've already seen Qwen's new QWQ 32B (not distilled) model doing impressive things on benchmarks. It'll definitely be interesting to see how just good small models can get. When combined with rag and large context window for expanded knowledge, might be able to get pretty far.