News for hackers. Stuff that matters.

A comment on An idea from physics helps AI see in higher dimensions. Parent comment

Re:An idea from physics helps AI see in higher dimensions by pavanky on Thursday January 09, 2020 @01:36PM (#22004031)
The weights arent averaged unless the training step is synchronous. Even then most of the times it is the gradients that are added up rather than the actual weights.

For inference, I dont think there are many papers that claim direct average of weights perform better than any single model. It is usually the output that is accumulated in some way.

[ Parent | on HN ]
Re:An idea from physics helps AI see in higher dimensions by sillysaurusx on Thursday January 09, 2020 @02:13PM (#22004407)
https://arxiv.org/abs/1803.05407

"Averaging Weights Leads to Wider Optima and Better Generalization"

Weirdly, the averaging doesn't have to be synchronous.

[ Parent | on HN ]