News for hackers. Stuff that matters.
Re:Beyond Gradient Averaging in Parallel Optimization by lassepe on Friday January 03, 2025 @09:35AM (#42585965)
Can you elaborate on if/when $\theta$ is synchronized across nodes?

Algorithm 1 suggests that each node starts gradient aggregation from their local micro-gradient $g$. Since the order of aggregation matters, \theta would likely diverge after apply the step with $g_{\mathrm{GAF}}$ --- even if models on different nodes are initialized with the same weights. Hence, I would expect there to be a weight-synchronization step after each macro-gradient step. Do you have such a step? If so, how do you implement consensus? Simply via averaging?

There are no comments on this comment yet.