Re:Regularization is all you need: simple neural nets can ex...
by jackylupino on Wednesday June 23, 2021 @06:29AM
(#27602866)
Batch norm has an advantage for iterative methods on mini-batches, while XGB uses the full training set. Using batch norm on the full training set is equivalent to Z-normalizing the features, which has no effect at all for XGB as the scale of features plays no role at the split decisions of the tree nodes. Apart few non-parametric data augmentation methods (notice adversarial augmentation is also nn specific), I do not think any other regularization used in that paper can be directly/intuitively applied to XGB.
SWA is pretty NN specific. So leave it out for XGB. There's a bunch that are relevant, and they could be very important.