Thompson/multi-armed bandit optimizes for outcome over the duration of the test, by progressively altering the treatment %. The test runs longer, but yields better outcomes while doing it.
It's objectively a better way to optimize, unless there is time-based overhead to the existence of the A/B test itself. (E.g. maintaining two code paths.)
A key point here is that P-Values optimize for detection of effects if you do everything right, which is not common as you point out.
> Thompson/multi-armed bandit optimizes for outcome over the duration of the test.
Exactly.
In particular, if you aren't doing perfectly random sampling it is meaningless. If you are concerned about other types of error than sampling error it is meaningless.
A significant p-value is nowhere near proof of effect. All it does is suggestively wiggle its eyebrows in the direction of further research.
By "effect" I mean "observed effect"; i.e. how likely are those results, assuming the null hypothesis.
If you are doing a lot of significance tests you need to adjust the p-level to divide by the number of implicit comparisons, so e.g. only accept p<0.001 if running ine test per day.
Alternatively just do thompson sampling until one variant dominates.