News for hackers. Stuff that matters.
Re:How to increase signups by 50% using “popup forms” by patio11 on Tuesday March 15, 2011 @09:20AM (#2327300)
I'm a bit troubled with my results being used to support that conclusion, in the matter of a pharmaceutical researcher who published "Penicillin is not an effective treatment for brain cancer" would be troubled to be cited in support of "Well, that tears it, medical research is a sham."

It seems to me that the issue isn't A/B testing, it is blind adoption of other people's recommendations. You certainly don't need A/B testing to give people suboptimal advice -- designers, developers, and marketers do it every day without any evidence whatsoever! ("Eat more bananas, they're good for brain cancer!" <-- the state of web design best practices and, ahem, not far off from the state of nutrition research)

Re:How to increase signups by 50% using “popup forms” by aristus on Tuesday March 15, 2011 @01:07PM (#2328368)
The same thing happens with optimizing code for speed/efficiency/etc. People want rules of thumb and they don't want to hear the truth, which is "Rules of thumb are usually bullshit. You need to measure, change something, then measure again. Your system may or may not be the same as mine."
Re:How to increase signups by 50% using “popup forms” by Tichy on Tuesday March 15, 2011 @10:55AM (#2327726)
Btw. I don't like your analogy, I did not conclude from your example that A/B Testing never works. But one example is enough to show that it does not always work. Specifically, the claim "use A/B Testing and your signup rate will improve x%" is apparently not true.
Re:How to increase signups by 50% using “popup forms” by Tichy on Tuesday March 15, 2011 @09:41AM (#2327378)
Of course, I have no doubts that A/B Testing is useful, I take more issues with the way it is being marketed. Likewise a lot of SEO is probably useful, but there are also a lot of black sheep.