Agreed. For neural networks. I would argue the opposite is true, you should just have a 16bit float that casts really large values to infinity silently without throwing errors, with a logistic lookup that maps "inf" to +/- 1... A mathematically incorrect float is operationally superior to the correct one.