This is not like we are asking GPT what is 2 times 3. It is not even a bug. Just that the author is using quantized multiplication which takes subset of floating point space and just makes all the value closest to nearest point. So 2 can be approximated as say 2.5 if there are few points. And that is not deterministic. Also it is a known thing that something like that could occur. It is just that neural network still seem to learn.