A few weeks ago that companies agent hacked another company to solve a task by whatever means possible. Where exactly are they overfitting for Safety here?

The overfitting is in the models they release to the public

It's one of those situations where it's too much and too little at the same time.

Their idea of safety is not letting you do "unsafe" things with their model, based on their definition and not yours. Silently sabotaging users should never have been a serious proposal but Anthropic almost went forward with it. They're also very aggressive about protecting their IP and now third parties too with the new restrictions against decompiling games, to the point many perfectly legal use cases are getting blocked.

But they never really cared about making sure their model does exactly what you want and nothing more, or that it's honest when it fails instead of trying desperate measures that end with a Hugging Face situation. If your agent decides to leak all your API keys, take down the company site and delete your personal files that's not their problem. They don't care about preventing it.

The second kind of safety is the one that actually matters in practice, but the big labs only seem to care about the first one.

I cannot handle it anymore. Its1000 tests for even the simplest of tasks. Im starting to feel like its just a way for it to save some compute by knowing if anything changed because if a testing apparatus instead of reading