One example is the whole IoT movement. Most of the real (industrial, medical, automotive) applications are efficiency/cost-savings focused. The central theme is connecting devices that we have now to better monitor, analyze, and utilize what we've already built.
Another example that is a bit further removed from computers is the whole Uber/car sharing trend. At any given time 95% of the cars in the world are parked somewhere, not in use. Even if we could only bring that down by 5%, we would be utilizing billions of dollars in capital more efficiently.
I also strongly believe that we will see this reflected in living situations with growing urban cores/hubs of activity and jobs. Suburbs are no longer economically viable at the scale at which we built and inhabited them post WWII, and as the infrastructure begins to decay and inequality increases, more and more people will move to the city to take advantage of the economies of scale of living a dense, vertical lifestlye.
I created a script that migrated data to a new system. The script took 48 hours to run, and we needed to do it over a weekend. The lead dev was unhappy with the performance. I suggested we just use a more powerful machine, and he gave me permission, but said it wouldn't help. I can't remember the specs of the machine, but it was double/triple as expensive as the original machine.
It improved performance only marginally. I went through my scripts, and started to debug it, looking for problems. Optimising my scripts easily doubled or tripled the speed.
Modern code generators are really, really good at straight-line code. Sure, if you're writing everything in Python or Ruby or PHP then there's a bunch on the table. But if you're already doing C or C++, probably not so much.
And going parallel / multithreaded / distributed is still hard.
I don't see low-hanging fruit unless you've already chosen to burn cycles for productivity. If you're already at the performance floor, all roads are going to be difficult.
The thing is it takes a ton of discipline to organize your data to get back that 10-50x and a lot of projects it just doesn't make sense.
C++ devs who work in high-performance industries (like games or finance) know how to write fast code. C++ is not used in most consumer software these days, and even if it is used, you often see slow practices like chasing pointers everywhere, using inheritance instead of generics, etc. Even in big-data, an industry that you'd think would really benefit from fast code, the primary open-source tools are in Java. You could probably recover a 2-4x speedup - two generations of Moore's Law - by rewriting Hadoop in C++, but nobody (besides Google, who is happily crunching data with MapReduce etc.) has bothered.
That's because writing fast, correct C++ code is hard, and it tends to be more economical to write in Python or Node.js and then rent more VMs. If it is no longer economical to rent more VMs, you can certainly recover a lot of that performance, but until we get languages that let you use the performance tricks of C++ but have the safety & ease of use of Python (I have high hopes for you, Rust), it probably won't happen.
I agree on all points, writing correct data oriented code is hard. I've got some hope for Rust though, a lot of the common patterns lend well to the pain points you enumerate(slices, pointer chasing, OO inheritance mess, ownership, etc).
Indeed but there is also huge scope for applying known techniques to existing languages, PHP7 for example is twice as fast as 5.6 (in the real world, I've seen this) with about half the memory usage.
The interesting thing is that anyone using PHP who goes to 7 gets that performance upgrade at essentially zero-cost.
In anything I'd written in the last 2 years I didn't have to change any code at all, just upgrade, run unit tests, do some sanity checks and deploy.
Performance increases was immediately noticeable on both response times and server loads.
One of the easiest wins I've ever had :).
Compilers will of course get better and better at automatically parallizing code, but a big thing that needs to occur in my opinion is the introduction of the concept of explicit partial ordering into mainstream programming languages. Traditional programming languages assume that the order statements occur in code is the order in which operations should occur (operator precedence for the generation of the AST excepted). Imagine instead a programming language where statements separated by a newline implied temporal equivalence, while statements separated by a semicolon implied a temporal ordering. This would enable the compiler to automatically parallelize code much more efficiently.
A bigger problem is that many algorithms aren't really parallelizable. Throw a single mutable shared variable into the mix - even something as innocuous as a counter, or memory allocation pointer, or a logging buffer - and suddenly you need locking and your threads may end up starving each other. Once of the biggest breakthroughs of MapReduce is that it forces you to write your algorithm as functions that operate on independent data. But as someone who's written a decent amount of MapReduces - the thought process about how you design a MapReduce-based algorithm is very different from (and significantly more difficult than) single-processor programming. You can't just go to an algorithms textbook and copy some pseudocode, you need to actually think through what the algorithm and what it's doing, and then rewrite many state updates as message-sends, merges, and CRDTs. (This, BTW, is why Google interviews focus so heavily on algorithms - in many other jobs "How would you find the shortest path between a trillion graph vertices?" is totally academic, but in big data it comes up all the time.)
Hardware will continue to evolve and since things can't stay the same and just get faster, things are going to get weirder. With that will come similarly weird low level APIs and c style language implementations (cuda, opencl, vulkan, spirv)
I still hold out hope that Rust, or some successor, will hit the sweet spot you mentioned.
There's a lot of low-hanging performance fruit in programming language design, compiler implementation, technology choice, and application design. For the most part, nobody takes it because it doesn't make economic sense. With Moore's Law, just wait a couple years and your shitty implementation is good enough.
It's possible for a skilled engineer to write code that is much more efficient. You have to compromise in other ways, like developer time or feature set. Up till now, those compromises have not been worth it, because somebody will beat you to market and make you irrelevant. In a world where "fast as an initial prototype" is not "fast enough", however, you could see firms actually start to compete on performance.
I predict that even without hardware changes, we'll see a bit of a maturation of the software industry, where people look at what folks actually do with their computers and then work to make that actually perform well. Possibly not for a few years, though.