I thought some NNs actually used fewer bits (32?) than the normal 64bit floating point because it is faster and good enough?

On PCs everybody uses 64 bits, and if they choose not to, they have a large risk of being converted into and out of it all the time by the compiler because it's faster.

But specialized CPUs often get different number sizes. It's common for specialized NN and GPU hardware to use 32 bits floats, what leads to 2^24 definition that is actually around the top precision you will find for analog components on the places people (instead of labs) go for buying them, at least on the countries where access to those aren't restrict due to nuclear proliferation concerns.

Floats on modern PCs are 32bit (you can look that up if you want). Doubles (double precision floats) are 64bit and are seperated from 32bit. There is also half precision float (16bit) since quite a few generations.

There is no conversion happening on the compiler level unless you choose to run with soft-floats for reasons unknown.

You don't need specialized GPU hardware for 32bit floats either; that is the default. You need specialized hardware if you want 64bit floats however (CAD Designers love those). Half precision is also available on all GPUs that people will likely care about.

Some NNs use 16bit floats instead of the normal 32bit floats, because yeah, it's good enough to cover what you need from low quality images for example (you get around 12bit color space, which can be enough for a lot of things)