Random Forests are nice in having few parameters-- number of trees, number of restricted features to sample on at each decision node.

It's the variations on the standard RF that have interested me. Rotation Forests, where features are partitioned randomly and rotated (by PCA or random projection) before decision boundaries are drawn. Extremely Randomized Forests, where the node splits are completely random and not based on best possible Gini/Entropy gain along some feature. There's even an interesting use of deterministic annealing out there for incorporating unlabeled data points in an attempt at semisupervised learning.

These each have their own parameters to tune, but have had slightly different performance on different problem domains. Even models require different data-- most decision trees have a quite natural way of imputing missing values, but something like a Rotation Forest can handle neither missing values nor categorical data (unless you map it to m binary features). And that complexity spooks me away from Machine Learning as a Service, where one could start failing to understand his or her models. (Plus then I'd probably be out of a job.)

I actually think that random forests are relatively difficult to tune in comparison to some Boosting methods for example.

The typical parameters are the following: - Number of trees - Percentage of data used to train each tree - Maximal depth of the tree - Minimum information gain (although this can usually be set to 0 and use only the depth) - Minimum sample size (same as the minimal information gain) - Number of thresholds to try for continuous data

Tuning is also difficult because of the non deterministic nature of the algorithm. If you compare to sets of parameters it requires more evaluation in order to be sure if one set is better because of the better choice of the parameters or because the algorithm chose the right data samples. This effect decreases with the number of trees, but the training time is increased in this case.

I think random forests are really good for very large datasets, but in my experience for smaller datasets boosting (for example JointBoost or GentleBoost) can give better results.

Speaking of bagging: Geoff Hinton talks a bit about model averaging and has a very stimulating pair of hypotheses related to this, which are:

a) genetic recombination in the form of sex might be "bagging across genes" that prevents tight co-adaptation (= overfitting on the evolutionary timescale)

b) the brain uses noisy discrete firing instead of continuous communication because it allows for "bagging across network topologies".

The bulk of his talk is about how dropout in neural nets can be used to accomplish some of the same performance benefits that model averaging gives on other algorithms (at a much smaller relative computational cost).

Here's the talk: https://www.youtube.com/watch?v=DleXA5ADG78

As textminer says, RFs are really nice in that there are few parameters to tune, and the results typically are not that sensitive to the choice of those parameters (contrasted to, say, SVMs, where you can get killed in performance with a poor choice of tuning parameters).

With the MLaaS platform, all of the model optimization is taken care of under the hood (we also allow users to do their own parameter selection / tuning if desired). Our super fast implementation, WiseRF (10-100x faster than RF in sklearn or R) enables us to efficiently explore the hyperparameter space.

Thanks again for your questions and comments.

Joey Richards, Chief Scientist, wise.io

Two more questions then; is this speed-up achieved on a single core or also due to parallelism? And have you compared WiseRF with Random Jungle (http://bioinformatics.oxfordjournals.org/content/26/14/1752....)?

(1) The speed-up is achieved on a single core (multithreaded). The tiny memory footprint enables us to do embedded learning (e.g., on an ARM chip). We also have a distributed version of WiseRF in development (stay tuned!).

(2) Soon, we'll be publishing a series of blog posts to benchmark WiseRF against competing implementations. Look for that next week.

Thanks for your interest!