As others commented, you either ensemble models (average the predictions) or average the update (gradient).

For ensembling, the mathematical justification for why this surprising result is true (e.g just averaging many weak but different models gives a better model) is pretty interesting: https://en.wikipedia.org/wiki/Condorcet%27s_jury_theorem