Re:Building a recommendation engine, foursquare style
by izendejas on Tuesday March 22, 2011 @10:39PM
(#2357689)
For the "coldplay" problem you may want to use an inverse band frequency weighting (much like in IDF for text). Bands that are extremely popular won't give you any useful signals. As far as the cold-start... yep, that's a beast and using social data may help. For example, are you using stuff from Facebook/Twitter once you have users sign in? You could crawl out band/artists name from their data.
This gives me an excuse to try grooveshark. I've been curious for a while. :)
update: I didn't see any recommendations based on what I've like on Facebook or Twitter. This would be a good "warm" start. Note: I'm working on stuff like this, but don't mind sharing this simple idea. :) Quora does a great job of this pulling in data from your Twitter network.
Re:Building a recommendation engine, foursquare style
by adpowers on Tuesday March 22, 2011 @10:37PM
(#2357685)
I'm curious: do you see any speedup from that Hadoop cluster? I always assume that on less than like 5 nodes it is faster to just run it all on a single machine unless you are highly disk bound.
Was it written that way in order to support future growth?
Re:Building a recommendation engine, foursquare style
by justinmoore on Wednesday March 23, 2011 @09:57AM
(#2359366)
Delis are our Coldplay.
40 machines!?! Wow. We compute our recommendations at Grooveshark in a couple hours on a crappy 2-node Hadoop cluster. hah. We have about as many songs as they have venues, so I have no idea why they need so many nodes. Or maybe we're just extraordinarily harsh in our constraints. I loved this writeup though. Its great to see how other companies tackle these difficult problems.
We also have that problem of a "cold-start" and another one which we call the "coldplay" problem. For them it would be something like McDonalds I guess.