Interesting idea to ensemble them. The main difference is the way the trees are constructed, right? Do ensembling them is kind of like saying "I don't know which one is better, so screw it I will do both an average the results", right?

Hey! CatBoost team here.

Yes, stacking different gradient boosting algorithms works well on practice. One example is a just finished kaggle-like competition: http://mlbootcamp.ru/round/12/sandbox/ where mpershin stacked CatBoost with LGBM and got the 7th place.

The kernel for this solution can be found in one of our tutorials: https://github.com/catboost/catboost/blob/master/catboost/tu...

if i am doing regressors, I use weights: like .7 xgb .3 lgb. I think its because they have their biases but the biasses are not completely correlated. This is one of the leading strategies in the kaggle competition for zillow logerror, check out this kernel: https://www.kaggle.com/aharless/xgb-w-o-outliers-lgb-with-ou...

When doing classification, I use several classifiers and get their scores rather than prediction via 20 fold xvalidation. I then use a random forest on top of the scores to get the final prediction. the random forest can figure out when to trust one classifier over another

Lesser predictors can actually help in ensembles, so long as their errors aren't highly correlated with the better predictors.