I agree that they are no panacea, but we also have an example of the Intel Xeon phi, which did have high speed hbm memory. That too did not compete with GPUs. I think the majority of it has to do with the GPU chip itself being extremely simple, with no Branch prediction, no branch difference, very simple caching behavior, etc. They are able to allocate a much larger portion of the die to just throw fma units at it.