It’s also true that people that perform their own evals see wide variance uncorrelated with “nerfing”. Human cognitive bias makes them only see that variance at specific times and then they start projecting.

Everyone really should keep a few tasks around that are indicative of the work they do and run them a few times with the same model and a few times across model families and when new models drop.

It’s extremely informative to see with your own eyes the variance that always exists. Even if you can’t afford statistical relevance it’s valuable.