Floats on modern PCs are 32bit (you can look that up if you want). Doubles (double precision floats) are 64bit and are seperated from 32bit. There is also half precision float (16bit) since quite a few generations.
There is no conversion happening on the compiler level unless you choose to run with soft-floats for reasons unknown.
You don't need specialized GPU hardware for 32bit floats either; that is the default. You need specialized hardware if you want 64bit floats however (CAD Designers love those). Half precision is also available on all GPUs that people will likely care about.
But specialized CPUs often get different number sizes. It's common for specialized NN and GPU hardware to use 32 bits floats, what leads to 2^24 definition that is actually around the top precision you will find for analog components on the places people (instead of labs) go for buying them, at least on the countries where access to those aren't restrict due to nuclear proliferation concerns.