Regularization is all you need: simple neural nets can excel on tabular data

Re:Regularization is all you need: simple neural nets can ex... by magicalhippo on Tuesday June 22, 2021 @09:34PM (#27599718)

> I have yet to see a real world production pipeline where GBDT provides enough improvement over Logistic Regression

Not my field at all, so "I know nooothing".

Are GBDT's very different from "plain" binary decision trees? I've seen the latter a lot in the context of particle experiments[1][2][3].

[1]: https://arxiv.org/abs/physics/0408124

[2]: http://cds.cern.ch/record/2289251/

[3]: https://arxiv.org/abs/2002.02534

[ Parent | on HN ]

Re:Regularization is all you need: simple neural nets can ex... by hogFeast on Wednesday June 23, 2021 @12:27AM (#27600788)

Very simply: plain decision trees usually overfit to training data (and, therefore, perform very badly out of sample). So the important part isn't the tree but the boosting. How you go from an ensemble of weak learners to something that works.

And this boosting generalises to any learner. You can apply it to regression too. Again, the boosting part is really the key. The innovation isn't a new technique either, it is just the aggressive application of computing power to these problems.

[ Parent | on HN ]

Re:Regularization is all you need: simple neural nets can ex... by jncraton on Tuesday June 22, 2021 @09:45PM (#27599812)

They are the same concept under the hood, but a GBDT is an ensemble model using a number of trees in tandem that are grown to improve the performance of the overall model.

[ Parent | on HN ]