Probably, the idea that the GPU handles the "hot neurons" means that if your GPU is strong enough to handle _enough_ hot neurons, you should expect speedups.