News for hackers. Stuff that matters.
Re:Beyond Gradient Averaging in Parallel Optimization by gwern on Tuesday December 31, 2024 @04:02PM (#42561803)
I'm also reminded of top-_k_ training/sampling, where you throw away the inconsistent samples: https://arxiv.org/abs/2002.06224 "Top-k Training of GANs: Improving GAN Performance by Throwing Away Bad Samples", Sinha et al 2020.

Which seems like it would be roughly equivalent to throwing away gradients if they are inconsistent...?

There are no comments on this comment yet.