News for hackers. Stuff that matters.
Re:A/B test improved your website's conversion rate? Not so ... by nrjames on Friday January 07, 2022 @08:45AM (#29839004)
My experience is similar. Even if and when the metrics are calculated properly, there's often some design or business reason put forth as an excuse to ignore them.
Re:A/B test improved your website's conversion rate? Not so ... by sosborn on Friday January 07, 2022 @10:59AM (#29840622)
Before any effort goes into something like this I always raise my hand and ask, "If the data shows us something we don't want to see, will we change our strategy? If not, I'd rather put time/effort into other projects." It works about 80% of the time.

Metrics are only useful if the organization is actually willing to learn lessons from them.

Re:A/B test improved your website's conversion rate? Not so ... by kqr on Friday January 07, 2022 @11:18AM (#29840916)
Oh yeah, defining the thresholds and their associated courses of action ahead of time is important to make a good decision.

Any time someone wants to measure something, the top two questions should be "what are the lower and upper bounds this value has to exceed for us to do something different?"

Very often, it turns out these thresholds for change are so astronomical that nobody thinks we have even the slightest chance of exceeding them. That means the measurement is completely useless. Whatever result we plausibly get, it won't change anything.

You can save a lot of time this way!

Re:A/B test improved your website's conversion rate? Not so ... by codeptualize on Friday January 07, 2022 @08:51AM (#29839071)
It seems that any kind of data is mainly used to confirm the believes people already had and/or justify decisions already made.
Re:A/B test improved your website's conversion rate? Not so ... by lewisl9029 on Friday January 07, 2022 @09:13AM (#29839287)
Not to mention the decision paralysis and change aversion it often introduces into company culture where every change, however trivial or however obviously beneficial, has to first go through a 2-week A/B test which often turns out to be inconclusive anyways, and sometimes takes more eng resources to set up and run than it takes to make the change itself.
Re:A/B test improved your website's conversion rate? Not so ... by ssharp on Friday January 07, 2022 @11:22AM (#29840985)
> however trivial

Previous testing should give the company at least some baseline understanding of what is trivial and what isn't. The correct way to experiment is certainly not "let's experiment every idea!"

> however obviously beneficial

If you've been around long enough, you've almost certainly run into dozens of "obviously beneficial" changes that led to poorer performance.

Most of what you're describing is issues with poor prioritization, a lack of understanding about your audience, and a culture that has a difficult time making decisions.