For solving linear systems there are recent mind-blowing methods by R. M. Gower and P. Richtarik:

https://arxiv.org/abs/1506.03296

http://www.maths.ed.ac.uk/~richtarik/papers/SDA.pdf

Plus R. M. Gower is fantastically nice and enthusiastic so there's that

And thanks for the link !

Only scrolled through it, but AFAIK if its results are comparable to Kaczmarz, then note that it is extremely slow.

If your goal is to solve say a LS problem, why not go for CGLS? http://web.stanford.edu/group/SOL/software/cgls/

Has anyone implemented them in production? I read their paper a while ago, but thought that the dual methods they showed might have practical limitations (like insane cache misses, compared to more standard methods)

These problems you sketch can already be overcome by using the approximate eigenspectrum info of the matrix obtained as a by-product of Krylov methods. This has been invented many times, and is used in 'thick restart', 'augmented Krylov subspace methods', 'deflation'. In particular this was developed for scattering problems with many electromagnetic incident waves hitting an object, changing only the rhs.

Also, you can also use an approximate LU-decomp as a preconditioner for Krylov methods.

Similarly, Tykhonov regularization (solving for a range of slightly perturbed matrices "A + labda I" where labda is a parameter) are easily tackled using Krylov subspace methods by noting that the Krylov subspace is invariant under shifts like these. So only once an orthonormal basis must be found for the Krylov subspace, which can then be used for every lamda of interest.