News for hackers. Stuff that matters.
Re:Beyond Gradient Averaging in Parallel Optimization by woadwarrior01 on Tuesday December 31, 2024 @07:22AM (#42558304)
Very interesting idea! Reminiscent of this robust learning paper[1] from 2020, where they do something even simpler: zero out elements in gradients with inconsistent signs, before averaging them.

[1]: https://arxiv.org/abs/2009.00329

Re:Beyond Gradient Averaging in Parallel Optimization by fchaubard on Tuesday December 31, 2024 @09:30AM (#42558880)
Hey thanks! Ya we tried similar strategies to this and could not beat cosine distance < tau, average, else skip. It was too much to put in the paper and we may put it in the arxiv version but we tried Sign AND gating and zero’ing out if the signs don’t agree, we tried L2<tau, etc but nothing beat cosine distance.
Re:Beyond Gradient Averaging in Parallel Optimization by gwern on Tuesday December 31, 2024 @04:02PM (#42561803)
I'm also reminded of top-_k_ training/sampling, where you throw away the inconsistent samples: https://arxiv.org/abs/2002.06224 "Top-k Training of GANs: Improving GAN Performance by Throwing Away Bad Samples", Sinha et al 2020.

Which seems like it would be roughly equivalent to throwing away gradients if they are inconsistent...?