News for hackers. Stuff that matters.
Re:The often-overlooked random forest kernel by stochastic_monk on Sunday October 29, 2017 @12:59PM (#15580079)
Very interesting; I've ignored random forests more than I should. Thank you!

I won't comment on your methods except to say that comparing classification over linear PCA against a kernel PCA in linearly unseparable data isn't exactly fair, and I think that providing an SVM performed on a different kernel PCA decomposition or a kernel SVM itself would be more illustrative. (Is your code available?)

I generally think of neural networks as enormous meta-kernels. (Composites of <composites of...> kernels) This generally leads me to think of ways that kernels can be turned into neural network layers.

Great work has been done turning powerful tools like Random Fourier Features/Kitchen Sinks into layers in neural networks (e.g., Alex Smola's Deep-Fried ConvNets [https://arxiv.org/abs/1412.7149] and Choromanski's Structured Adaptive/Random Spinners[https://arxiv.org/abs/1610.06209]). Deep Forest [https://arxiv.org/abs/1702.08835] is a method which claims to work well, but it's somewhat odd; It's not quite what I would have imagined.

My biggest criticism of random forests is that the more expressive models are more memory-hungry and expensive at both training-time and run-time than many comparable methods. But bounded-complexity trees with smart implementations seem to be a lot more useful and have broader applications than I give them credit for.

Re:The often-overlooked random forest kernel by RMarcus on Sunday October 29, 2017 @03:58PM (#15581071)
I agree the comparison could be a lot more fair. My goal was to show how adding in labels can give you better principle components, but I should've used an RBF kernel, or some kind of unsupervised kernel, instead of linear PCA.

I really should go and make the code available. I spent a lot of time writing the RF kernel in Cython to get it to be fast. But it is so fragile and incomplete, I'm not sure I am ready to release and maintain it. Maybe I'll just post the scripts.

Thanks for the links and insights about ANNs as combined kernel learners + classifiers. I'll check out the papers (I'm also surprised at what Deep Forest is. Not what I was expecting.)

Re:The often-overlooked random forest kernel by theSage on Friday November 03, 2017 @07:06AM (#15618202)
What were you expecting deep forests to be like?