Re:Beyond Gradient Averaging in Parallel Optimization
by fchaubard on Tuesday December 31, 2024 @09:30AM
(#42558880)
Hey thanks! Ya we tried similar strategies to this and could not beat cosine distance < tau, average, else skip. It was too much to put in the paper and we may put it in the arxiv version but we tried Sign AND gating and zero’ing out if the signs don’t agree, we tried L2<tau, etc but nothing beat cosine distance.
Re:Beyond Gradient Averaging in Parallel Optimization
by gwern on Tuesday December 31, 2024 @04:02PM
(#42561803)
I'm also reminded of top-_k_ training/sampling, where you throw away the inconsistent samples: https://arxiv.org/abs/2002.06224 "Top-k Training of GANs: Improving GAN Performance by Throwing Away Bad Samples", Sinha et al 2020.
Which seems like it would be roughly equivalent to throwing away gradients if they are inconsistent...?
[1]: https://arxiv.org/abs/2009.00329