Re:Netflix never used its $1 million algorithm due to engine...
by benhamner on Friday April 13, 2012 @07:50PM
(#3839329)
As Eliezer said - they implemented the two most important algorithms from the contest, which advanced the state of the art in recommendation systems & gave the majority of the benefits, and didn't implement the long tail of algorithms that each only gave a very slight marginal benefit & would have been costly to re-train and maintain.
This is one of the advantages of running shorter competitions: normally it takes 1-3 months to approximately hit the asymptotic level of performance on a dataset given the inherent noise in it and the state of the art in machine learning. The shorter competitions are focused on finding the low-hanging fruit that generate large improvements (such as SVD & RBM's in Netflix's case) and exploring the space of possible model structures, as opposed to optimally ensembling across a large number of models to eek out the last 0.01% of performance.
Exploring the space of useful features & possible models enables you to trade off computational efficiency & maintainability vs. model performance in production as well. The $1 million dollars Netflix put to the prize leveraged >> $1 million in human effort to explore the possible models, from which they found and applied the two best suited for their production implementation.
This is one of the advantages of running shorter competitions: normally it takes 1-3 months to approximately hit the asymptotic level of performance on a dataset given the inherent noise in it and the state of the art in machine learning. The shorter competitions are focused on finding the low-hanging fruit that generate large improvements (such as SVD & RBM's in Netflix's case) and exploring the space of possible model structures, as opposed to optimally ensembling across a large number of models to eek out the last 0.01% of performance.
Exploring the space of useful features & possible models enables you to trade off computational efficiency & maintainability vs. model performance in production as well. The $1 million dollars Netflix put to the prize leveraged >> $1 million in human effort to explore the possible models, from which they found and applied the two best suited for their production implementation.
(disclaimer - I work with Kaggle)