Very large models trained with sgd variants don't tend to over fit, especially when using few epochs. See double descent.

The coding/math breakthroughs were caused by training models on the subset of their own outputs which pass a verifier, see reinforcement learning from verified rewards.