You are averaging weights in distributed training? That seems like it would be rife with pitfalls unless you average after every batch.

I always thought the preferred method was to average the gradient updates, and pass that to update the single mother-model.