Re:A/B testing gets misused to juice metrics in the short term
by ssharp on Tuesday July 12, 2022 @09:34AM
(#32069673)
It's not really an either/or option. You can use testing to validate the changes stemming from market research.
Having seen lots of site redesigns go horribly wrong due to 100% earnest people trying their best and utilizing the research that was afforded to the process, I always recommending incrementally testing into changes on high-value / high-risk applications, even when the "improvements" were backed by solid research. You never know until you release.
Re:A/B testing gets misused to juice metrics in the short term
by mschuster91 on Tuesday July 12, 2022 @10:56AM
(#32070917)
The "or" was meant to be the distinction on who does the user testing - I've seen both in-house testing operations and outsourced ones. For small scale operations, it may actually be cheaper to run them in-house and only hire external testers... cameras are dirt cheap these days.
Hiring a market research company is usually worth it if you have a contract with them anyway (which gives you better rates on the testing) or lack someone on staff who knows how to deal with cameras.
For that you usually hire a market research company or do what they will do: take an interviewer, two cameras (one front-face, one top-hands) and hire an as-diverse-as-possible pool of test candidates that you then put through whatever workflow optimization you want to do. Then afterwards, you interview them - side benefit, you can get really interesting general side knowledge that you'd never gain from a dumbass A/B scheme: is your font style/color scheme legible, can the site be used by colorblind people, are there stock photo choices that give off stereotypical vibes...
It's real fun and a worthwhile experience for everyone involved.