Not really - since GPUs have started to hit the same "Process" size as CPUs - and haven't been showing a lot of growth in that area. The best improvement per Wikipedia's chart is a small foray into 14 and 12nm, and those haven't doubled the transistor counts (per square mm).

What we are seeing is an increase in die sizes; more parallel cores. Parallel cores still require parallel algorithms, so I stand by my earlier statement.

Latest NVidia GPUs are ~1.3-1.4 times faster, than previous gen with the same or lower power consumption.

Moore's law is about transistor count and not performance. Performance of those parallel architectures always get subject to Amdahl's rule of scaling.

This is partly why we see slowing scaling in performance and why you cannot just throw cores at the problem.

You're conflating things here; Moore's law is about how many transistors you can pack onto a chip, not performance. Moore's law is unsustainable due to laws of physics. Transistors are reaching size scales where quantum effects like tunneling dominate.

Think of a transistor as having two regions, separated by a channel. When the transistor is on, charge carriers flow through the channel between the regions. When off, charge carriers do not flow. But when we move to smaller and smaller length scales, the channel is so small that charge carriers will tunnel through and reach the other region. How will you distinguish on/off behavior now?

Todays transistors are still generally on a plane. If we could find ways to build IC's stacked thousands or millions of layers thick, we could really start using the third dimension.

("3d" transistors built on their side don't really count)

For CPUs/GPUs it probably wouldn't help that much, since you have to get rid of the heat produced by switching a transistor.

And for storage the main issue isn't how big it is, but how cheap it is to manufacture.

Maybe this is silly, but could you use redundancy and statistics to help?