> Remark. It is important to acknowledge that previous research has highlighted the ability of neural networks to tolerate noise and even act as a regularization method. However, our proposed analysis differs in two ways. Firstly, it is the first comparison of its kind that focuses on pure 16-bit neural networks. Secondly, our analysis offers a unique quantification approach that distinguishes it from previous explanations related to regularization. Our approach examines floating-point errors and formalizes the concept of tolerance, allowing for its quantification as a measure of comparison with floating-point errors.

> Explanation of the Lemma.... in the worst case.... But as long as... the two classifiers M16 and M32 must have the same classification result on x.

So my rough understanding is that they're saying "Hey, it's not/not only because of regularization; it's because fp16 is really as good as fp32 for what we need it to do (but more efficient?"

This is sort of my party trick I do in a number of places in diagnosing NNs from scare environmental data, let me give it a whirl. It sounds like your network has far too many degrees of freedom. Reducing it via residuals or some other method will likely help with that.

Additionally, some L2 weight decay, switching to SGD+OneCycle, don't forget to BatchNorm before every activation as well.

If this is a newer-style attention-Unet ala StyleGAN then that would be a confusing result as transformers seem to be pretty okay with not immediately collapsing to that kind of thing if I understand correctly.

Barring all of that, swapped labels can be a surprising reason for complex data to overfit as it forces the network into a memorization-only mode with very little chance for generalization.

Let me know if I got it correct/close for you. :) :D <3 :))))