News for hackers. Stuff that matters.

A comment on Ask HN: Can someone ELI5 Transformers and the “Attention is all we need” paper. Parent comment

Re:Ask HN: Can someone ELI5 Transformers and the “Attention ... by throwawaymaths on Friday May 19, 2023 @09:17AM (#36001575)
It actually isn't. You have to have a synchronizer, batchsize one or else "strange things" can happen and you waste a lot of cycles. Alternatively you can do non-simple changes to your network structure to enable distributed training.
[ Parent | on HN ]
Re:Ask HN: Can someone ELI5 Transformers and the “Attention ... by Salgat on Friday May 19, 2023 @11:18AM (#36002984)
It really is that simple. Yes, there's many different approaches to this (which can become quite clever and complex, which is true of training in general), but it all really boils down to adding or averaging the gradients in most cases.
[ Parent | on HN ]