At least from my experience with sparse matrix libraries like Eigen, you need to get the sparsity down to about 5% before switching from a dense to a sparse algorithm gets you execution time benefits.

Of course from a memory bandwidth and model size perspective maybe there are benefits long before that.