Re:A/B test improved your website's conversion rate? Not so ...
by kqr on Friday January 07, 2022 @11:18AM
(#29840916)
Oh yeah, defining the thresholds and their associated courses of action ahead of time is important to make a good decision.
Any time someone wants to measure something, the top two questions should be "what are the lower and upper bounds this value has to exceed for us to do something different?"
Very often, it turns out these thresholds for change are so astronomical that nobody thinks we have even the slightest chance of exceeding them. That means the measurement is completely useless. Whatever result we plausibly get, it won't change anything.
You can save a lot of time this way!
Metrics are only useful if the organization is actually willing to learn lessons from them.