Re:Beyond Gradient Averaging in Parallel Optimization
by gwern on Tuesday December 31, 2024 @04:02PM
(#42561803)
I'm also reminded of top-_k_ training/sampling, where you throw away the inconsistent samples: https://arxiv.org/abs/2002.06224 "Top-k Training of GANs: Improving GAN Performance by Throwing Away Bad Samples", Sinha et al 2020.
Which seems like it would be roughly equivalent to throwing away gradients if they are inconsistent...?
Which seems like it would be roughly equivalent to throwing away gradients if they are inconsistent...?