Re:Google’s look, evolved
by nostrademons on Tuesday September 01, 2015 @05:40PM
(#10155947)
There are plenty of talented people working on ranking and webspam too.
There's a limit to how parallelizable those problems are. At some point, more people working on an algorithm doesn't make it faster or better, it just makes it more confusing and worse. Mythical Man Month applies as much to Google as it did to IBM. And so you might as well put people onto optimizing the rest of the page as well, because they'll do more good there than having yet another cook in the ranking algorithm kitchen. (I'd argue that there are too many people working at Google in general...I left, so I'm no longer contributing to that problem. But then, there were probably too many people when I joined, so I contributed to it for 5 years.)
The worst limitation of A/B testing is that it's not producing anything better than a local maxima. Nature does indeed work like that, but it took millions of years along with random mutations and we don't have that much time. And if in our thinking process we would have relied on optimization algorithms favoring the local maxima, then Jazz wouldn't have been produced, amongst others.
And most A/B testing focuses on the very short time effects of the change (and change all by itself, even change for the worse can have a short term positive effect just because something is new). On another note: A/B testing has another limitation, which is that you're optimizing for the bulk, please one person more and you more or less automatically annoy another.