Cross-entropy loss can start getting worse due to the model becoming less calibrated, even as the classification accuracy continues to improve. I first heard that here: https://arxiv.org/abs/1706.04599

Is this 'overconfidence' the leading explanation as to why LLMs continue to show qualitative improvement even after their test loss levels off?