Re:Most AB-tests will fail
by janesvilleseo on Wednesday June 19, 2013 @02:15PM
(#5907209)
I agree with you above statement. However, in regards to your example. I often wonder how much of the increase is due to Google overcoming ad blindness and how much is because of the ads are truly more engaging.
Good question. That's where longer-term measurement comes in. Overcoming ad blindness is a transitory thing that goes away after a period of time while engagement tends to stick (as measured by CTR, say).
I'm sensitive to hyperbole, and I know he later refines his statement, but this is not true. Google Ads change what seem to be insignificant details by tiny amounts all the time (luminance on links, vertical space below the search bar, padding around sitelinks) and derive significant revenue changes.
I think one issue is that most sites' A/B tests results are difficult to distinguish from noise while Google has what you might call a significant user base.