depends how often you need to synchronize. look at deep learning, you need to sync the weights as often as possible