Why do tree-based models still outperform deep learning on tabular data?

Re:Why do tree-based models still outperform deep learning o... by thom on Wednesday August 03, 2022 @08:04PM (#32337872)

What are the principled ways that tree based models handle categorical data? If you end up having to do one-hot encoding it feels like you need very wide forests or very deep trees. If your categorical data is actually vaguely continuous then splits can be quite efficient but that’s rare.

I assume some day someone will be able to explain all this in information theoretic terms. I’m never sure if we’re comparing like with like (are the deep learning models we’re comparing against actually that deep, for example?) but clearly there’s something to the intuition that many small overfit models are more efficient than one big general model.

[ Parent | on HN ]

Re:Why do tree-based models still outperform deep learning o... by CapmCrackaWaka on Wednesday August 03, 2022 @09:05PM (#32338244)

Some implementations will group the categories into two groups, right and left. An easy way to do this is to create the groups based on whatever grouping creates the biggest decrease in the loss. There are more complicated implementations, lightgbm groups by the sum(gradient) / (sum(hessian) + smooth) where smooth is some smoothing parameter which prioritizes class size over loss decrease.

They link to this paper in their docs: https://www.tandfonline.com/doi/abs/10.1080/01621459.1958.10...

[ Parent | on HN ]

Re:Why do tree-based models still outperform deep learning o... by coffee_am on Friday August 05, 2022 @08:24AM (#32355084)

One way is to use categorical set splits [1] (proposed for categorical set inputs, but works for categorical features as well), used in TF-DF [1]. Greedy, and expensive to train (cheap inference though), but it gives great results.

[1] https://arxiv.org/pdf/2009.09991.pdf [2] https://www.tensorflow.org/decision_forests/text_features

[ Parent | on HN ]