Bigger L1/L2 caches is definitely another "easy" thing with substantial benefits, but doing it cheaply is hard.

Those caches are SRAM which has not improved much the last few generations.

That's not easy at all because larger caches are slower. A higher hit rate with worse latency can give worse performance. We may see more levels of cache to better optimize the tradeoffs.

You mean moving memory closer to compute. And that's happening already in lots of places, even though it brings increased complexity due to having to account for NUMA.

I am thinking long term, as neural nets scale up, we'll have to move compute into memory. The other way around is problematic, small caches don't work well with neural nets. In fact the fastest Transformer (FlashTransformer) is based on principled usage of the SRAM cache because that's the bottleneck.