Re:Out of the cesspool and into the sewer: A/B testing trap
by samlittlewood on Monday June 21, 2010 @11:04AM
(#1448999)
It is worth looking at "Simulated Annealing":
http://en.wikipedia.org/wiki/Simulated_annealing
Start your optimization process with a high "temperature" (leading to big changes), and let it "cool" over subsequent iterations.
Re:Out of the cesspool and into the sewer: A/B testing trap
by paraschopra on Monday June 21, 2010 @12:14PM
(#1449213)
See http://visualwebsiteoptimizer.com/split-testing-blog/how-aqu... as an example of testing radical change.
See http://visualwebsiteoptimizer.com/split-testing-blog/using-a... as an example of subtle change.
In my opinion, it is entirely misguided to blame local maxima problem on A/B testing. The methodology doesn't dictate what is that you want to test. It is all about having a hypothesis and then collecting data to prove/disprove it. The post looks like it is blames poor results on instruments while wrong experimental design is the true culprit.
A very powerful concept that you don't see done very often.