Look up the double descent phenomenon. Large models with lots of data generalize rather than overfit.
Also see the argument around figure 3 here. https://arxiv.org/pdf/2503.02113
Also see the argument around figure 3 here. https://arxiv.org/pdf/2503.02113
Is there any other mechanism replacing it? As far as I understood, while following the LLMs from Scratch book, the dropout is a mechanism to prevent overfitting, and it makes a lot of sense (basically cripple some neuron connections randomly during training such that their weights aren't updated).
Edit: well, I could have searched on the internet, but other people may stumble on your post and you seem quite knowledgeable of the inner workings of these things. :)