News for hackers. Stuff that matters.

A comment on Netflix Progress Prize Awarded. Parent comment

Re:Netflix Progress Prize Awarded by jderick on Wednesday December 10, 2008 @08:02PM (#393739)
The machine learning guys have only a subset of the full dataset to work with. They have to send their algorithms to Netflix for testing on the full dataset. So it may be possible Netflix knows something they don't. It does seem quite strange how close and yet how far that 10% mark has been.
[ Parent | on HN ]
Re:Netflix Progress Prize Awarded by Retric on Wednesday December 10, 2008 @08:50PM (#393792)
Because they are testing it vs a fixed data set teams can gain information about that data set based on their submissions. Which is why several teams mix and mach several different approaches. They can increase the weight of each approach over time and game the system.

Edit: I am not saying they are intentionally cheating rather it's one of the reasons that approach works.

[ Parent | on HN ]
Re:Netflix Progress Prize Awarded by pg on Wednesday December 10, 2008 @08:51PM (#393794)
Hmm. What could Netflix know, though? Is it even possible to quantify how hard it is to improve a recommendation algorithm by a certain amount?
[ Parent | on HN ]
Re:Netflix Progress Prize Awarded by aswanson on Wednesday December 10, 2008 @10:17PM (#393888)
Maybe there is an inherent noise floor for the optimal estimator for the dataset they have found or bounded.
[ Parent | on HN ]