News for hackers. Stuff that matters.
Re:Why small tweaks and split testing don't work, and what t... by hn_throwaway_99 on Thursday November 29, 2018 @02:45PM (#18563275)
Sorry to be so harsh, but this article is mostly BS. Booking.com built there entire corporate culture around A/B testing, and they (i.e. parent Priceline) were one of the top performing stocks of the 00s.

If anything, the biggest issue I've seen with A/B testing is that it biases organizations to things that are easy to measure (i.e. shopping conversion) while sometimes at the expense of things that are longer term/harder to measure (like brand reputation). I'd be most interested to hear how some companies have dealt with that shortcoming.

But all that said, one of the biggest benefits of a culture around A/B testing is that it gets companies away from the endless back-and-forth around opinions (like this article) and builds a culture of "OK, we'll try your suggestion and see if it succeeds or fails".

Re:Why small tweaks and split testing don't work, and what t... by pedrocortes on Thursday November 29, 2018 @02:51PM (#18563319)
No problem ;)

If you noticed I included a section explaining when a split test makes sense and you can tell that Booking.com fit it that criteria as they were already starting to grow a lot.

Re:Why small tweaks and split testing don't work, and what t... by hammock on Thursday November 29, 2018 @02:53PM (#18563338)
Hi, I highly doubt booking.com's success came from CTA button colors - or any testing on the consumer side of the platform, for that matter. Their success is rooted in how they are able to take advantage of hotels and their distribution power.
Re:Why small tweaks and split testing don't work, and what t... by hn_throwaway_99 on Thursday November 29, 2018 @03:09PM (#18563480)
This is absolutely false, and anyone who works at booking.com will tell you this. https://taplytics.com/blog/how-booking-comss-tests-like-nobo...
Re:Why small tweaks and split testing don't work, and what t... by hammock on Thursday November 29, 2018 @06:02PM (#18564990)
Because they test a lot doesn't mean it's critical to their success. That's a cargo cult fallacy.

Your article points to 2-3x industry average conversion rates (for existing traffic), but says nothing about the more important factors of acquisition or inventory.

Re:Why small tweaks and split testing don't work, and what t... by hn_throwaway_99 on Thursday November 29, 2018 @08:39PM (#18565873)
Except that Booking actually has data to back this all up. It's very common to do "holdbacks" so that you, say, give an old version of your website to a very small portion of your user's, then compare its conversion rate to a version of your site that has all the latest A/B test winners. Booking also tracks religiously how conversion rates change over time.

The whole point being that the way Booking (and other) companies act is the exact opposite of a cargo cult. Everything needs to be backed up with data and challenging conclusions is part of the culture.

All that said, a common complaint of Booking is that it has a ton of "dark UI" patterns, so it will be interesting if there is any long term blowback against short term A/B winners that erode the goodwill to the brand over time.