A naive GPU implementation on a non-block sparse network will be just as slow as the full un-pruned network.

Another approach would be to lower precision. By going from 32bits to 16 or 8 you can get massive speedups in memory operations and computation.

Nvidia's apex.amp does just this - attempts half-precision training where possible, and adjusts scaling factors during training. I achieved a very significant speedup using this on my RTX 2080 Super, though it doesn't support Windows that well.

Yes, that'll work. But it's orthogonal to this paper.

The trick is to combine these 2 approaches. Quantizing to bytes helps a lot, but forces you to use much larger pruning group size to see any speed up from pruning.