Floating point arithmetic is a significant source of bugs. And if you transition to fixed point arithmetic, you will still have a significant source of same-same-but-different bugs (maybe unless you really only need to add things and not never scale or multiply them).
What kind of bugs do you have in mind for fixed point?
While there no doubt would be some, I have a feeling they are a significantly lesser drag (in frequency and debugging complexity) than floating point bugs. Assuming the language supports fixed point.
Of course you can imagine error-prone fixed point too. Writing shaders with plain integers while trying to manually keep track heterogenous fixed point scales would be a nightmare for example...
CPUs and GPUs do have support for fixed point (aka integer) data formats, it's been a DX/OpenGL requirement for a while. Though I don't know what API issues you run into if you try to avoid FP altogether.