Regarding the second comparison, it's easy to do trivially better than someone else's work if you use their work as a starting point. But that type of optimization doesn't really matter as much as the first one you mentioned: the ability to "take that which would have run on a CPU, and parallelize it to run on a GPU.". It has always felt quite natural for me to do that, so it was strange to hear that it's so hard for others. Why is it difficult for them, but easy for me?

The easiest way for me to solve this mystery is to try to port an example someone considers difficult. Do you know of any?

try matrix multplication. I mean a real matrices, say 10000x10000. And once you're there, try a LU decomposition of matrix, with proper numerical erro handling. You'll soon get to the point where you fell it's harder than you tought. And BTW, matrix multiplication or LU decomposition are super fundamental stuff when solving equations (which in turn is super fundamental when you want to compute bridges, constructions, aerodynamics, etc...)

Sparse or dense matrices?

I don't know very much about this, but do you think that part of your success has just been the benefit you get from going from dynamic ram to static ram? Seems that alone would give you a baseline performance improvement?

GPUs use dynamic RAM too (though discrete cards usually have GDDR rather than DDR, which gives higher bandwidth). The most fundamental difference between a CPU and GPU isn't the memory hierarchy (modern GPUs are growing cache hierarchies as well) but the core architecture (vector/highly threaded "small cores", vs. out-of-order/superscalar "big cores").