Doesn't a lot of optimization target making your code and data more cache friendly because memory latency (not bandwidth?) kills performance absolutely (between other things like port usage I guess)?

If something is in L3 it is better for CPU "utilization" than stalling and reaching out to RAM. I guess there are eventually diminishing returns with too much cache, but...