Cache friendly algorithms tend to be on the order of 10-50x speed improvement so they're definitely worth it, the problem is that once you've already built your app you've already locked yourself into an architecture that can't be changed enough to take advantage of them.

Dig up some of posts around "Data oriented design" from the gamedev space if you want to see it done proper.