The thing is it takes a ton of discipline to organize your data to get back that 10-50x and a lot of projects it just doesn't make sense.
C++ devs who work in high-performance industries (like games or finance) know how to write fast code. C++ is not used in most consumer software these days, and even if it is used, you often see slow practices like chasing pointers everywhere, using inheritance instead of generics, etc. Even in big-data, an industry that you'd think would really benefit from fast code, the primary open-source tools are in Java. You could probably recover a 2-4x speedup - two generations of Moore's Law - by rewriting Hadoop in C++, but nobody (besides Google, who is happily crunching data with MapReduce etc.) has bothered.
That's because writing fast, correct C++ code is hard, and it tends to be more economical to write in Python or Node.js and then rent more VMs. If it is no longer economical to rent more VMs, you can certainly recover a lot of that performance, but until we get languages that let you use the performance tricks of C++ but have the safety & ease of use of Python (I have high hopes for you, Rust), it probably won't happen.
I agree on all points, writing correct data oriented code is hard. I've got some hope for Rust though, a lot of the common patterns lend well to the pain points you enumerate(slices, pointer chasing, OO inheritance mess, ownership, etc).
Modern code generators are really, really good at straight-line code. Sure, if you're writing everything in Python or Ruby or PHP then there's a bunch on the table. But if you're already doing C or C++, probably not so much.
And going parallel / multithreaded / distributed is still hard.
I don't see low-hanging fruit unless you've already chosen to burn cycles for productivity. If you're already at the performance floor, all roads are going to be difficult.