hendurhance

I mean this should be expected, models learn from us, and "WE" game the metrics time after time. I also don't think it will just disappear just because we clean pretraining data. I believe the deeper reason is optimization, if you point any optimizer at a proxy objective it finds the cheapest path to the number, whether or not the corpus ever contained "examples of cheating."

And it can't be a prompt-level fix because it is like telling an optimizer "don't take that shortcut", it's just more constraints for it to go around toward the same objective.

Most people use divs for everything now. This is a breath of fresh air.

I understand the appeal due to the speed

I feel like, since 4.0, it is pretty much the same model but with new names. They are just improving the CoT and function calling.