Some of the weird new quants really do push the limits, but at the end of the day that compute you're eliminating is the actual 'work' the model is doing.

Folks like to think of 'weights' as just storage, but they aren't (except with stuff like engrams). Those digits of precision encode actual computation, not just storage. There is always a cost, it's lossy compression after all. It's like MP3 vs AAC. If you can't hear the difference it just means you haven't encountered it YET.

That's not to say that you ever will. The cost could be in weird far out edge cases, but whatever it once encoded... It's gone now and it ain't coming back.