Re:Lessons Netflix Learned from the AWS Storm
by justinsb on Friday July 06, 2012 @06:57PM
(#4210007)
Are recommendations computed in real-time though? Have you considered e.g. batch recomputation overnight with a 'full' algorithm, and just applying a linearized model to any newly rated content?
I feel like the quality of the Netflix recommendations is not stellar, and if that's because you're constraining yourself to what can be calculated in real-time, I'd willingly trade-off having "perfect" real-time recommendations in favor of better recommendations tomorrow (with the full model). Even if you do try to update recommendations in real time, aren't they easily cacheable if you can't keep up? (Well, as easily cacheable as any dataset on 25 million subscribers can be...)
http://techblog.netflix.com/2012/04/netflix-recommendations-...
In parts one and two, it's explained that just about every list generated for users is an up-to-date set of recommendations. You point out yourself that recommendations is one area where caching it's truly viable (at least, not in a traditional sense).
I'm definitely not the right person to go into all the details (nor do I think such a discussion would be prudent on HackerNews)--but I wanted to weigh in quickly that there's a lot of stuff served that goes way beyond the notion of a "static" content that's trivially cached.