There are literally performance breakthroughs being made every few weeks (most recently, PowerInfer, which can speed up LLM inference by 10x or more) where the main improvement amounts to the application of caching/preloading techniques.

If the "engineering" part of ML had kept up with the "science" part, I have no doubt that performance for typical use cases would be 100x-1000x higher than we are seeing today.