Exactly right. I feel like anyone with a comprehensive eval suite (companies running their own agents) should see this signal and it would be a well known fact by now if that were the case.

The only thing I could maybe see is that any changes that are being made are happening on the subscription tiers on not the api tiers, but even that seems like quite a stretch.

Like anyone who is building agents with a good eval suite would know, models have been pretty consistent and even small harness changes can impact performance considerably.