GBDT have their own set of hyperparameters such as learning rate, number of trees, min samples per bin, l0, l1, etc. So you could definitely also create an appropriate cocktail to optimize on, although GBDT are typically more robust wrt huperparameters.

The authors do claim to do a hyperparameter sweep but only for vanilla XGB hyperparams.

The old "my method (with as much optimization as I could get away with) beats the other method (with as little optimization as I could get away with)"

Yup. The Least Publishable Unit strikes again.

Batch normalization is nothing neural network specific to it if you use it on the input layer. I don't think it matters for a tree algorithm like XGBoost either way though.

SWA is pretty NN specific. So leave it out for XGB. There's a bunch that are relevant, and they could be very important.

Batch norm has an advantage for iterative methods on mini-batches, while XGB uses the full training set. Using batch norm on the full training set is equivalent to Z-normalizing the features, which has no effect at all for XGB as the scale of features plays no role at the split decisions of the tree nodes. Apart few non-parametric data augmentation methods (notice adversarial augmentation is also nn specific), I do not think any other regularization used in that paper can be directly/intuitively applied to XGB.