w2tanh(w1x) = -1w2tanh(-w1*x)
But if you average the weights in those two equivalent models you get 0.
If you're talking about asynchronous data parallelism, then there can be some averaging of weights, but they all start with the same weights and are re-synched often enough that weights are never too different to break it.
We average the weights themselves, and the efficiency seems to be similar to gradient gathering.
It’s also averaging in slices, not the full model. There’s never a full resync.
SWA is the theoretical basis for why it works, I think.
Another way of thinking about it: If the gradients can be averaged, then so can the weights.
What works is averaging similar networks and averaging your networks a lot of times.
http://jmlr.csail.mit.edu/papers/volume17/16-137/16-137.pdf
The math there is above my paygrade, but it describes a way to structurally combine a certain class of ML models for very significant performance gains. More importantly, it describes why it works.
I always thought the preferred method was to average the gradient updates, and pass that to update the single mother-model.
If you think about what sort of brain you'd get if you "averaged" a few hundred geniuses' brains by blending them into a soup and adding some gelatin to a brain-sized sample, yeah, that's about the level of intelligence I saw in the "average neural net." It was basically in a constant state of seizure.
I think this is one of the least strange things in AI. All you're doing is taking N overfitted models (unlikely to be overfit in the same way) and then asserting that the average of those predictions is probably not overfitted as much (regularization). Overfitting as a concept is not restricted to some number of dimensions.
It doesn't really matter though, because the parent comment is nonsense and you can't just average the weights at the end and get a working neural net.
For inference, I dont think there are many papers that claim direct average of weights perform better than any single model. It is usually the output that is accumulated in some way.
"Averaging Weights Leads to Wider Optima and Better Generalization"
Weirdly, the averaging doesn't have to be synchronous.
For ensembling, the mathematical justification for why this surprising result is true (e.g just averaging many weak but different models gives a better model) is pretty interesting: https://en.wikipedia.org/wiki/Condorcet%27s_jury_theorem
This is how distributed training often works, for example. Data parallelism.
I don’t understand why it still works in higher dimensions, but it seems to.
The intuition is that the multiple models are “spinning” around the true solution, so averaging gives the final result more quickly. But it works even early in the training process.