News for hackers. Stuff that matters.
Re:Why Do Random Forests Work? They Are Self-Regularizing Ad... by math_dandy on Sunday October 20, 2024 @12:00AM (#41892756)
This is good intuition for why ensembling overparametrized is a good idea. Doesn’t speak to why ensembles of tree-structured estimators in particular perform so well compared to ensembles of other nonparametric estimators.
Re:Why Do Random Forests Work? They Are Self-Regularizing Ad... by youoy on Sunday October 20, 2024 @03:03AM (#41893498)
If you look at what makes it work well in the example, I would say it is being able to easily approximate a function with whatever degree of precision that you want, which translates to being able to isolate spikes in the approximation.

For example, one could ask, what if instead of an approximation by step functions, we use a piecewise linear approximation (which is as good)? You can do that with a fully connected artificial neural network with ReLU nonlinearity, and if you check it experimentally, you will see that the results are equivalent.

Why do people often use ensembles of tree structures? The ensembling part is included in the programming packages and that is not the case for ANN, so it is quicker to experiment with. Appart from that, if you have some features that behave like categorical variables, trees also behave better in training.