I think that if you survey the researchers in the field, the "WTF test" would be considered fairly reasonable - especially for a quick-and-dirty evaluation. Can you point to any specific songs that he said were WTF's and you think aren't, or vice/versa? If not then it would appear to meet the objectivity criteria.

Using his own music collection might be slightly more suspect. Changing that might have flipped the outcome of iTunes vs EchoNest, but wouldn't have changed the real news here: Google does really, really badly.

You couldn't do a test with other collections because the beta is very limited right now-- I can only think of a few people I know that have access. But I can concur that my results are as terrible as his were.

Yes, he works for EN (which he's very clear about) and yes it's a bit inside baseball showing a service that most people can't use (because they're not developers or customers) but you can scroll past our results & take it as a post titled "How is Apple so much better than Google at such a data driven task?"

(I would have given the lack of beatles on genius -24 too! I also found a couple EN clunkers that I'd give a WTF to, but I'm a notorious jerk for those things, ask anyone i work with :)

Sure, but that wouldn't exactly be fair seeing as Genius is 3+ years mature, and Google Music+Instant Mix is in beta and less than a week old. OP even seems to think so...

> The last time I took a close look at iTunes Genius was 3 years ago. It was generating pretty poor recommendations.

- Genius was much better than this when it launched. It's even better now, but it wasn't as bad back then as Instant Mix is today.

- You don't think google has more data about music than Apple did when it launched Genius? YouTube, Music Onebox, search traffic.

- (severe bias alert) EN's playlist APIs are the same excellent quality today as the day it launched (sept '10.) We're roughly 0.10% Google's size. We didn't need any warm up period.

Just curious, in regard to the quality of EN since launching.. You guys we're clearly working on your algorithm for several years prior to launch[1] in late 2010.. That's sort of a warm-up period, no?

[1] second last paragraph of http://blogs.oracle.com/plamere/entry/genius_or_savant_syndr... from 2008

Why do they think it is so important to get in this market ASAP? Perhaps under Page there is additional pressure to launch fast and early a la start-ups?

With that said, I have no idea if the quality of EN is as good as he claims.

http://techcrunch.com/2008/03/27/first-machine-listening-api...

Re: "warmup": My point was technologies like ours don't need a warm up period where users give us catalog and preference data. You can have great results without relying on that. Even better, we think.

Also, re: "please buy us" -- come by the office sometime and have some drinks and you'll see why we all found that comment pretty funny. Hopefully someone still reading this thread knows us well enough to +1 this. Come to a music hack day if you're not near boston.

(post-edit: davemebs -- our stuff is pretty easy to try for yourself if you're a developer. see developer.echonest.com . if not, wait a little bit for some consumer facing things. we're primarily developer-focused at the moment)

..so? I'd honestly like to see more of this sort of advertising.

While the author could have been more objective about his criteria, ironically enough, I think he missed the more salient point that he raised by implication: that music engines should be mapping the user's behavior patterns vis-a-vis the songs in his collection, and not so much objective connections between songs. This is what Genius does and has always tried to do, and it's why Genius seems to work better for the author. Genius doesn't focus as heavily on attempts to forge objective links between songs, so much as it focuses on attempts to draw links in behavior patterns w/r/t songs by likeminded users.

When listening to songs in a collection, our brain maps out its own connections between songs, as reflected in the way we compile our own lists consciously or subconsciously. Sometimes those connections make objective sense (i.e., "I want to listen to '70s funk, so I'm going to pick ten '70s funk songs in a row."). Sometimes those connections make little objective sense (i.e., "I am listening to a track by Lady Gaga, and afterward, I feel like listening to a track by J.S. Bach."). A good mixing engine figures out the idiosyncracies and subjectivity of our brains, as reflected statistically by the choices we've made in the past.

There's no objective evaluation of playlists. I've proposed a simple, subjective one that I think gets the job done. I'm happy to try other ones if you have something to suggest.

From my perspective, I'd say an empty playlist is worth 24 WTFs, in the sense of "WTF, I expected 24 similar songs and got 0!"

Would you rather be given no results or bad results? I agree that no results isn't good but I'd rather an application that actually identified it couldn't do a good job than one that just threw a load of junk at me.

I think a score of 50% or so is probably about fair. It shouldn't get a good score certainly but I'd certainly rate it higher than one which just produced nonsense.