News for hackers. Stuff that matters.
Re:Regularization is all you need: simple neural nets can ex... by nightcracker on Tuesday June 22, 2021 @04:23PM (#27596713)
Batch normalization is nothing neural network specific to it if you use it on the input layer. I don't think it matters for a tree algorithm like XGBoost either way though.

SWA is pretty NN specific. So leave it out for XGB. There's a bunch that are relevant, and they could be very important.

Re:Regularization is all you need: simple neural nets can ex... by jackylupino on Wednesday June 23, 2021 @06:29AM (#27602866)
Batch norm has an advantage for iterative methods on mini-batches, while XGB uses the full training set. Using batch norm on the full training set is equivalent to Z-normalizing the features, which has no effect at all for XGB as the scale of features plays no role at the split decisions of the tree nodes. Apart few non-parametric data augmentation methods (notice adversarial augmentation is also nn specific), I do not think any other regularization used in that paper can be directly/intuitively applied to XGB.