Nvidia's apex.amp does just this - attempts half-precision training where possible, and adjusts scaling factors during training. I achieved a very significant speedup using this on my RTX 2080 Super, though it doesn't support Windows that well.

Yes, that'll work. But it's orthogonal to this paper.

The trick is to combine these 2 approaches. Quantizing to bytes helps a lot, but forces you to use much larger pruning group size to see any speed up from pruning.