Actually, nope! SGD is embarrassingly parallel when it's HOGWILD! https://arxiv.org/abs/1106.5730. Formally, the hogwild and serial implementations are indistinguishable. This has a lot of citations but it's such an easy trick that no door path beating seems to have happened. The "trick" is just to make model updates atomic, and then each SGD step can happen in parallel without coordination or locks.